Jul 31 23:46:09.412538 ip-10-0-140-138 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 23:46:09.849927 ip-10-0-140-138 kubenswrapper[2546]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:46:09.849927 ip-10-0-140-138 kubenswrapper[2546]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 23:46:09.849927 ip-10-0-140-138 kubenswrapper[2546]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:46:09.849927 ip-10-0-140-138 kubenswrapper[2546]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 23:46:09.849927 ip-10-0-140-138 kubenswrapper[2546]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 23:46:09.852084 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.851996    2546 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 23:46:09.856890 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856875    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:09.856890 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856889    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856893    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856897    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856900    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856903    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856908    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856912    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856915    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856918    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856921    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856924    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856927    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856930    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856932    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856935    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856938    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856940    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856943    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856953    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:09.856956 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856956    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856959    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856962    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856964    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856967    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856970    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856973    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856976    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856978    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856981    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856983    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856987    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856990    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856992    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856995    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.856997    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857000    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857003    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857005    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:09.857413 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857007    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857011    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857015    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857018    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857020    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857023    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857025    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857028    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857031    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857033    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857036    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857038    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857041    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857044    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857046    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857049    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857052    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857054    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857057    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:09.857875 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857060    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857062    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857064    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857067    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857070    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857072    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857075    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857077    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857080    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857082    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857085    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857087    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857090    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857093    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857096    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857098    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857101    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857103    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857105    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857108    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:09.858387 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857110    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857113    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857115    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857119    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857121    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857123    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857127    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857132    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857136    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857139    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857142    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857145    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857147    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857150    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857152    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857155    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857157    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857160    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857163    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:09.858861 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857165    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857168    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857865    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857872    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857875    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857880    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857884    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857887    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857891    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857893    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857896    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857898    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857901    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857904    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857906    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857909    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857911    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857914    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857917    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857919    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:09.859335 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857923    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857925    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857928    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857930    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857933    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857936    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857940    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857943    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857946    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857948    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857951    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857953    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857956    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857958    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857961    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857963    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857965    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857968    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857971    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:09.859825 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857973    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857976    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857979    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857982    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857984    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857987    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857989    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857991    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857994    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857996    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.857999    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858001    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858004    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858007    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858010    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858013    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858015    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858018    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858022    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:09.860295 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858026    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858029    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858032    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858035    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858037    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858040    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858043    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858046    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858048    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858051    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858053    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858056    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858058    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858062    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858064    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858067    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858070    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858072    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858075    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858077    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:09.860754 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858079    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858082    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858084    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858087    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858089    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858091    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858095    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858097    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858100    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858103    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858106    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858109    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858112    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858114    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858116    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858119    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858121    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858124    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858127    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858129    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:09.861235 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858132    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858135    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858137    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858516    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858522    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858525    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858528    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858531    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858534    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858537    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858539    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858542    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858546    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858550    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858553    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858556    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858559    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858564    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858567    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:09.861749 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858570    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858572    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858575    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858578    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858581    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858584    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858587    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858589    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858592    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858594    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858597    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858599    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858602    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858604    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858607    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858609    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858612    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858614    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858617    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858619    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:09.862211 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858621    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858624    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858627    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858630    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858633    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858635    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858638    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858640    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858643    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858646    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858649    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858652    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858654    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858657    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858659    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858662    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858664    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858667    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858670    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858672    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:09.862718 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858675    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858677    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858680    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858683    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858685    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858688    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858690    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858693    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858695    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858698    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858702    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858705    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858708    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858712    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858715    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858718    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858721    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858724    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858728    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:09.863232 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858732    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858734    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858737    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858746    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858749    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858751    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858754    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858756    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858759    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858761    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858764    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858767    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858769    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858772    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858774    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858777    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858780    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858782    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858784    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858787    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:09.863705 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858790    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858792    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858795    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.858797    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858859    2546 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858866    2546 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858875    2546 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858879    2546 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858883    2546 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858886    2546 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858890    2546 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858895    2546 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858898    2546 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858901    2546 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858904    2546 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858908    2546 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858911    2546 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858914    2546 flags.go:64] FLAG: --cgroup-root=""
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858917    2546 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858920    2546 flags.go:64] FLAG: --client-ca-file=""
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858922    2546 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858925    2546 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858929    2546 flags.go:64] FLAG: --cluster-domain=""
Jul 31 23:46:09.864190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858932    2546 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858935    2546 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858938    2546 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858942    2546 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858945    2546 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858948    2546 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858951    2546 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858954    2546 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858957    2546 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858960    2546 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858963    2546 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858966    2546 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858969    2546 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858973    2546 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858976    2546 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858980    2546 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858983    2546 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858987    2546 flags.go:64] FLAG: --enable-server="true"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858990    2546 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858995    2546 flags.go:64] FLAG: --event-burst="100"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.858998    2546 flags.go:64] FLAG: --event-qps="50"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859001    2546 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859004    2546 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859007    2546 flags.go:64] FLAG: --eviction-hard=""
Jul 31 23:46:09.864755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859011    2546 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859013    2546 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859016    2546 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859019    2546 flags.go:64] FLAG: --eviction-soft=""
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859022    2546 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859025    2546 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859028    2546 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859031    2546 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859034    2546 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859037    2546 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859040    2546 flags.go:64] FLAG: --feature-gates=""
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859043    2546 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859047    2546 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859050    2546 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859053    2546 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859056    2546 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859059    2546 flags.go:64] FLAG: --help="false"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859061    2546 flags.go:64] FLAG: --hostname-override="ip-10-0-140-138.ec2.internal"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859064    2546 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859067    2546 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859071    2546 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859074    2546 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859077    2546 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859082    2546 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 23:46:09.865346 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859085    2546 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859087    2546 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859090    2546 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859093    2546 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859097    2546 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859099    2546 flags.go:64] FLAG: --kube-reserved=""
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859102    2546 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859105    2546 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859108    2546 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859125    2546 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859129    2546 flags.go:64] FLAG: --lock-file=""
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859132    2546 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859136    2546 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859139    2546 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859145    2546 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859148    2546 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859151    2546 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859154    2546 flags.go:64] FLAG: --logging-format="text"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859157    2546 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859160    2546 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859163    2546 flags.go:64] FLAG: --manifest-url=""
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859166    2546 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859170    2546 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859173    2546 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859177    2546 flags.go:64] FLAG: --max-pods="110"
Jul 31 23:46:09.865951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859179    2546 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859182    2546 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859185    2546 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859189    2546 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859192    2546 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859195    2546 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859199    2546 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859205    2546 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859208    2546 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859211    2546 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859214    2546 flags.go:64] FLAG: --pod-cidr=""
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859217    2546 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859222    2546 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859225    2546 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859228    2546 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859231    2546 flags.go:64] FLAG: --port="10250"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859234    2546 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859237    2546 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c07d2146a831b80c"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859241    2546 flags.go:64] FLAG: --qos-reserved=""
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859244    2546 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859247    2546 flags.go:64] FLAG: --register-node="true"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859263    2546 flags.go:64] FLAG: --register-with-taints=""
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859272    2546 flags.go:64] FLAG: --registry-burst="10"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859275    2546 flags.go:64] FLAG: --registry-qps="5"
Jul 31 23:46:09.866620 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859278    2546 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859281    2546 flags.go:64] FLAG: --reserved-memory=""
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859285    2546 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859288    2546 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859291    2546 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859294    2546 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859297    2546 flags.go:64] FLAG: --runonce="false"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859300    2546 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859303    2546 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859306    2546 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859309    2546 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859312    2546 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859315    2546 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859318    2546 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859321    2546 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859325    2546 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859328    2546 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859330    2546 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859333    2546 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859336    2546 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859340    2546 flags.go:64] FLAG: --system-cgroups=""
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859343    2546 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859348    2546 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859353    2546 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859356    2546 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 23:46:09.867193 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859360    2546 flags.go:64] FLAG: --tls-min-version=""
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859363    2546 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859365    2546 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859368    2546 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859371    2546 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859374    2546 flags.go:64] FLAG: --v="2"
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859378    2546 flags.go:64] FLAG: --version="false"
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859382    2546 flags.go:64] FLAG: --vmodule=""
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859386    2546 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.859389    2546 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859495    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859499    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859502    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859505    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859508    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859511    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859514    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859517    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859520    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859522    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859525    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:09.867815 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859527    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859531    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859533    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859536    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859539    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859541    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859544    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859547    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859549    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859554    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859556    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859559    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859561    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859564    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859566    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859569    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859571    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859574    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859576    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859579    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:09.868347 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859582    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859584    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859587    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859589    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859591    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859594    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859596    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859599    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859603    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859606    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859609    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859612    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859615    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859618    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859621    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859624    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859627    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859630    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859632    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:09.868829 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859635    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859637    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859642    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859644    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859647    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859650    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859652    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859655    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859657    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859660    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859663    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859665    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859668    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859670    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859673    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859676    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859678    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859681    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859683    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859687    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:09.869292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859691    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859693    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859696    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859699    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859702    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859704    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859707    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859710    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859713    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859715    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859718    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859721    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859724    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859727    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859731    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859734    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859736    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859739    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859741    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859745    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:09.869782 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859749    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:09.870275 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859751    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:09.870275 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859754    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:09.870275 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859757    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:09.870275 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859760    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:09.870275 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859762    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:09.870275 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859765    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:09.870275 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859767    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:09.870275 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.859769    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:09.870275 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.860686    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:46:09.870275 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.867667    2546 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 23:46:09.870275 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.867681    2546 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 23:46:09.870275 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.868548    2546 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 23:46:09.870275 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.868555    2546 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 23:46:09.870275 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868611    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868618    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868622    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868626    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868629    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868632    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868635    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868638    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868641    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868644    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868647    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868650    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868653    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868657    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868660    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868663    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868666    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868668    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868671    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868674    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:09.870695 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868677    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868680    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868683    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868686    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868689    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868691    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868694    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868697    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868699    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868703    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868706    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868709    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868711    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868714    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868717    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868719    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868722    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868724    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868727    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:09.871179 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868730    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868733    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868735    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868738    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868740    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868743    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868746    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868749    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868752    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868754    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868757    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868759    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868762    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868765    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868767    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868770    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868772    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868775    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868777    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868780    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:09.871660 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868782    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868785    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868787    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868790    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868793    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868795    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868798    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868801    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868803    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868807    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868810    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868813    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868816    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868819    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868822    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868824    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868827    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868830    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868833    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868837    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:09.872170 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868840    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868843    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868846    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868848    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868851    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868854    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868856    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868859    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868861    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868864    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868866    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868869    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868871    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868874    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868876    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868880    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868884    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868886    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868889    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:09.872760 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.868892    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:09.873227 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.868897    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:46:09.873227 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869008    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 23:46:09.873227 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869013    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 23:46:09.873227 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869016    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 23:46:09.873227 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869019    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 23:46:09.873227 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869022    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 23:46:09.873227 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869024    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 23:46:09.873227 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869027    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 23:46:09.873227 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869029    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 23:46:09.873227 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869032    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 23:46:09.873227 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869035    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 23:46:09.873227 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869037    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 23:46:09.873227 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869041    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 23:46:09.873227 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869043    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 23:46:09.873227 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869046    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869049    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869051    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869053    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869056    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869059    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869061    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869064    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869066    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869069    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869071    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869074    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869076    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869079    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869081    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869084    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869087    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869089    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869092    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869095    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 23:46:09.873618 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869097    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869100    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869102    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869105    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869107    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869110    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869112    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869115    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869118    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869121    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869123    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869126    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869129    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869131    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869134    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869137    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869139    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869142    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869144    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869147    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 23:46:09.874102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869150    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869152    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869155    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869158    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869160    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869163    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869165    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869168    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869171    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869173    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869176    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869178    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869181    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869183    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869186    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869188    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869190    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869193    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869196    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 23:46:09.874590 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869198    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869201    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869204    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869206    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869209    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869212    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869216    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869218    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869221    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869225    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869228    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869231    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869234    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869237    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869240    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869242    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869245    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869247    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 23:46:09.875050 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869262    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 23:46:09.875574 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869266    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 23:46:09.875574 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869269    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 23:46:09.875574 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869272    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 23:46:09.875574 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869275    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 23:46:09.875574 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869278    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 23:46:09.875574 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869280    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 23:46:09.875574 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869283    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 23:46:09.875574 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:09.869286    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 23:46:09.875574 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.869290    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 23:46:09.875574 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.869382    2546 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 23:46:09.875574 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.871724    2546 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 23:46:09.875574 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.873377    2546 server.go:1019] "Starting client certificate rotation"
Jul 31 23:46:09.875574 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.875227    2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 23:46:09.875574 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.875342    2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 23:46:09.878598 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.878580    2546 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 23:46:09.882749 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.882733    2546 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 23:46:09.894930 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.894911    2546 log.go:25] "Validated CRI v1 runtime API"
Jul 31 23:46:09.897282 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.897270    2546 log.go:25] "Validated CRI v1 image API"
Jul 31 23:46:09.898907 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.898851    2546 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 23:46:09.901474 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.901450    2546 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d8a2b4be-22be-4b41-904a-de0132e9e988:/dev/nvme0n1p4 f21790ed-bc3d-4bfb-91e4-af7ca4dde543:/dev/nvme0n1p3]
Jul 31 23:46:09.901548 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.901473    2546 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 23:46:09.902393 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.902370    2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 23:46:09.906676 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.906567    2546 manager.go:216] Machine: {Timestamp:2026-07-31 23:46:09.906090775 +0000 UTC m=+0.365553579 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097042 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2171835abee6a1645068acf5f196bd SystemUUID:ec217183-5abe-e6a1-6450-68acf5f196bd BootID:be36666a-5f4d-4e99-89d5-b8a7e83008da Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f0:e8:a5:fa:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f0:e8:a5:fa:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:70:aa:98:79:12 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 23:46:09.906676 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.906663    2546 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 23:46:09.906821 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.906755    2546 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 23:46:09.907976 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.907950    2546 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 23:46:09.908101 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.907977    2546 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-138.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 23:46:09.909468 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.909458    2546 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 23:46:09.909501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.909471    2546 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 23:46:09.909501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.909484    2546 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 23:46:09.910299 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.910289    2546 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 23:46:09.911506 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.911496    2546 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 23:46:09.915588 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.915577    2546 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 23:46:09.915707 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.915697    2546 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 23:46:09.915743 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.915726    2546 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 23:46:09.915743 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.915737    2546 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 23:46:09.915821 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.915747    2546 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 23:46:09.915821 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.915755    2546 kubelet.go:398] "Adding apiserver pod source"
Jul 31 23:46:09.915821 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.915764    2546 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 23:46:09.916107 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.916096    2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 23:46:09.916138 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.916119    2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 23:46:09.919707 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.919680    2546 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 23:46:09.921668 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.921645    2546 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 23:46:09.921742 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.921676    2546 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 23:46:09.923202 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.923189    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 23:46:09.923247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.923204    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 23:46:09.923247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.923210    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 23:46:09.923247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.923216    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 23:46:09.923247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.923225    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 23:46:09.923247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.923230    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 23:46:09.923247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.923238    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 23:46:09.923247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.923243    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 23:46:09.923247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.923262    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 23:46:09.923485 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.923269    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 23:46:09.923485 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.923278    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 23:46:09.923485 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.923283    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 23:46:09.923485 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.923302    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 23:46:09.923485 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.923308    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 23:46:09.924883 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.924864    2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2dcf8"
Jul 31 23:46:09.926177 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.926159    2546 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-138.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 23:46:09.926269 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:09.926173    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 23:46:09.926269 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.926221    2546 server.go:1268] "Started kubelet"
Jul 31 23:46:09.926269 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:09.926233    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 23:46:09.926421 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.926283    2546 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 23:46:09.926421 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.926335    2546 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 23:46:09.926506 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.926454    2546 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 23:46:09.927049 ip-10-0-140-138 systemd[1]: Started Kubernetes Kubelet.
Jul 31 23:46:09.927990 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.927976    2546 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 23:46:09.928327 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.928312    2546 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 23:46:09.930731 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.930716    2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2dcf8"
Jul 31 23:46:09.932042 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:09.932024    2546 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 23:46:09.936187 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.936169    2546 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 23:46:09.936187 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.936184    2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 23:46:09.936334 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:09.936227    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:09.936941 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.936922    2546 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 23:46:09.937008 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.936959    2546 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 23:46:09.937008 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.936923    2546 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 23:46:09.937118 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.937020    2546 factory.go:55] Registering systemd factory
Jul 31 23:46:09.937118 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.937036    2546 factory.go:223] Registration of the systemd container factory successfully
Jul 31 23:46:09.937118 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.937082    2546 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 23:46:09.937118 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.937092    2546 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 23:46:09.937309 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:09.936091    2546 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-138.ec2.internal.18c78498ff24f73c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-138.ec2.internal,UID:ip-10-0-140-138.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-138.ec2.internal,},FirstTimestamp:2026-07-31 23:46:09.9261991 +0000 UTC m=+0.385661909,LastTimestamp:2026-07-31 23:46:09.9261991 +0000 UTC m=+0.385661909,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-138.ec2.internal,}"
Jul 31 23:46:09.937309 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.937290    2546 factory.go:153] Registering CRI-O factory
Jul 31 23:46:09.937309 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.937303    2546 factory.go:223] Registration of the crio container factory successfully
Jul 31 23:46:09.937420 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.937351    2546 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 23:46:09.937420 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.937373    2546 factory.go:103] Registering Raw factory
Jul 31 23:46:09.937420 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.937384    2546 manager.go:1196] Started watching for new ooms in manager
Jul 31 23:46:09.937691 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.937680    2546 manager.go:349] Starting recovery of all containers
Jul 31 23:46:09.946398 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.946378    2546 manager.go:354] Recovery completed
Jul 31 23:46:09.948189 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.948166    2546 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:09.952280 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.952267    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:09.952692 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:09.952668    2546 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-138.ec2.internal\" not found" node="ip-10-0-140-138.ec2.internal"
Jul 31 23:46:09.954550 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.954533    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:09.954638 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.954564    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:09.954638 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.954578    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:09.955115 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.955102    2546 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 23:46:09.955115 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.955114    2546 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 23:46:09.955226 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.955130    2546 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 23:46:09.955226 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.955139    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:09.955803 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.955789    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:09.955844 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.955816    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:09.955844 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.955826    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:09.956118 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.956106    2546 policy_none.go:49] "None policy: Start"
Jul 31 23:46:09.956155 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.956126    2546 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 23:46:09.956155 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.956136    2546 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 23:46:09.957117 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.957106    2546 policy_none.go:47] "Start"
Jul 31 23:46:09.984439 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.984427    2546 manager.go:337] "Starting Device Plugin manager"
Jul 31 23:46:09.984504 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:09.984452    2546 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 23:46:09.984504 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.984462    2546 server.go:87] "Starting device plugin registration server"
Jul 31 23:46:09.984681 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.984667    2546 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 23:46:09.984731 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.984684    2546 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 23:46:09.984778 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.984763    2546 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 23:46:09.985004 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.984849    2546 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 23:46:09.985004 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:09.984862    2546 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 23:46:09.985267 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:09.985240    2546 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 23:46:09.985369 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:09.985354    2546 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:10.068705 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.068577    2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 23:46:10.069775 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.069755    2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 23:46:10.069830 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.069798    2546 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 23:46:10.069830 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.069821    2546 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 23:46:10.069911 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:10.069859    2546 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 23:46:10.072687 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.072666    2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:10.085014 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.084995    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:10.085762 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.085744    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:10.085829 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.085778    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:10.085829 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.085791    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:10.085829 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.085810    2546 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.096292 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.096279    2546 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.096339 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:10.096299    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-138.ec2.internal\": node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:10.110959 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:10.110904    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:10.170447 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.170419    2546 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-138.ec2.internal"]
Jul 31 23:46:10.170519 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.170492    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:10.171765 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.171747    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:10.171830 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.171773    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:10.171830 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.171787    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:10.173221 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.173210    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:10.173350 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.173338    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.173401 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.173364    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:10.173844 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.173828    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:10.173904 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.173847    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:10.173904 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.173856    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:10.173904 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.173864    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:10.173904 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.173868    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:10.173904 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.173874    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:10.174877 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.174866    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.174936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.174887    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 23:46:10.175466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.175453    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 23:46:10.175536 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.175473    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 23:46:10.175536 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.175486    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeHasSufficientPID"
Jul 31 23:46:10.199772 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:10.199754    2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-138.ec2.internal\" not found" node="ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.203848 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:10.203834    2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-138.ec2.internal\" not found" node="ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.211002 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:10.210989    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:10.239017 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.238996    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e68ec58a1ffc2594df23e7ea19433036-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal\" (UID: \"e68ec58a1ffc2594df23e7ea19433036\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.239073 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.239022    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e68ec58a1ffc2594df23e7ea19433036-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal\" (UID: \"e68ec58a1ffc2594df23e7ea19433036\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.239073 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.239038    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d1004082fdc42266a4ca7100481dac3f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-138.ec2.internal\" (UID: \"d1004082fdc42266a4ca7100481dac3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.311196 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:10.311176    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:10.339537 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.339516    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e68ec58a1ffc2594df23e7ea19433036-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal\" (UID: \"e68ec58a1ffc2594df23e7ea19433036\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.339592 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.339484    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e68ec58a1ffc2594df23e7ea19433036-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal\" (UID: \"e68ec58a1ffc2594df23e7ea19433036\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.339592 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.339576    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e68ec58a1ffc2594df23e7ea19433036-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal\" (UID: \"e68ec58a1ffc2594df23e7ea19433036\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.339658 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.339593    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d1004082fdc42266a4ca7100481dac3f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-138.ec2.internal\" (UID: \"d1004082fdc42266a4ca7100481dac3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.339658 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.339616    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d1004082fdc42266a4ca7100481dac3f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-138.ec2.internal\" (UID: \"d1004082fdc42266a4ca7100481dac3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.339658 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.339638    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e68ec58a1ffc2594df23e7ea19433036-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal\" (UID: \"e68ec58a1ffc2594df23e7ea19433036\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.411790 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:10.411735    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:10.502361 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.502332    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.506437 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.506373    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-138.ec2.internal"
Jul 31 23:46:10.512195 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:10.512178    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:10.612667 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:10.612641    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:10.715585 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:10.715562    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:10.821018 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:10.820997    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:10.874347 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.874330    2546 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 23:46:10.874705 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.874463    2546 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 23:46:10.874705 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.874463    2546 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 23:46:10.921245 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:10.921193    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:10.936397 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.936368    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 23:41:10 +0000 UTC" deadline="2028-04-21 05:39:20.686604338 +0000 UTC"
Jul 31 23:46:10.936455 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:10.936395    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15101h53m9.750214046s"
Jul 31 23:46:11.022508 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:11.022478    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:11.078242 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.078216    2546 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:11.123015 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:11.122990    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:11.219109 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:11.219081    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68ec58a1ffc2594df23e7ea19433036.slice/crio-9c6eedd26d22854fc88ee85fc52bec0f03537b7ae3908d40f1c8fc9dc6353a90 WatchSource:0}: Error finding container 9c6eedd26d22854fc88ee85fc52bec0f03537b7ae3908d40f1c8fc9dc6353a90: Status 404 returned error can't find the container with id 9c6eedd26d22854fc88ee85fc52bec0f03537b7ae3908d40f1c8fc9dc6353a90
Jul 31 23:46:11.223302 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:11.223287    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:11.225485 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.225467    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:46:11.324836 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:11.324810    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:11.425300 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:11.425248    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-138.ec2.internal\" not found"
Jul 31 23:46:11.493242 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.493189    2546 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:11.536743 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.536714    2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal"
Jul 31 23:46:11.551348 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.551331    2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 23:46:11.552156 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.552143    2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-138.ec2.internal"
Jul 31 23:46:11.561653 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.561634    2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 23:46:11.702761 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:11.702727    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1004082fdc42266a4ca7100481dac3f.slice/crio-1925a2ef67a9ade2966ce80338ab22d3b2ec83f8e7e9413902094d128c739c23 WatchSource:0}: Error finding container 1925a2ef67a9ade2966ce80338ab22d3b2ec83f8e7e9413902094d128c739c23: Status 404 returned error can't find the container with id 1925a2ef67a9ade2966ce80338ab22d3b2ec83f8e7e9413902094d128c739c23
Jul 31 23:46:11.919869 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.919787    2546 apiserver.go:52] "Watching apiserver"
Jul 31 23:46:11.928383 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.928353    2546 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 23:46:11.930320 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.930289    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-l94lx","openshift-ovn-kubernetes/ovnkube-node-2qmq9","kube-system/global-pull-secret-syncer-zmcw2","kube-system/kube-apiserver-proxy-ip-10-0-140-138.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr","openshift-cluster-node-tuning-operator/tuned-9bmfk","openshift-dns/node-resolver-v58mz","openshift-image-registry/node-ca-hp4zj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal","openshift-multus/multus-additional-cni-plugins-4q4d7","kube-system/konnectivity-agent-bdpjh","openshift-multus/multus-hjsqg","openshift-multus/network-metrics-daemon-5qcqv","openshift-network-diagnostics/network-check-target-wr59z"]
Jul 31 23:46:11.932142 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.932119    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:11.933618 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.933437    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:11.933618 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:11.933556    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:11.934943 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.934920    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 23:46:11.935058 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.934924    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 23:46:11.935058 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.935029    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:11.935173 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:11.935117    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:11.935234 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.935207    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 23:46:11.935316 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.935241    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 23:46:11.935316 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.935266    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 23:46:11.936275 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.936240    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pbqsg\""
Jul 31 23:46:11.936555 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.936429    2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 23:46:11.936555 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.936450    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:11.936555 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:11.936517    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:11.937704 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.937684    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:11.938910 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.938890    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:11.940242 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.940207    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hp4zj"
Jul 31 23:46:11.940378 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.940326    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j86rj\""
Jul 31 23:46:11.940464 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.940390    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 23:46:11.940574 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.940552    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 23:46:11.940718 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.940698    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 23:46:11.941287 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.941249    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:11.941543 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.941525    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:11.942153 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.941713    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9mk2s\""
Jul 31 23:46:11.942153 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.941954    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v58mz"
Jul 31 23:46:11.942947 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.942902    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 23:46:11.943086 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.943065    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-shzh7\""
Jul 31 23:46:11.943191 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.943073    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 23:46:11.943263 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.943240    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:11.943370 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.943349    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 23:46:11.944622 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.944593    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 23:46:11.944785 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.944762    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 23:46:11.944876 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.944859    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xs4q6\""
Jul 31 23:46:11.944986 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.944967    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l94lx"
Jul 31 23:46:11.945855 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.945835    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 23:46:11.945952 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.945916    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 23:46:11.946132 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.946050    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bmfrb\""
Jul 31 23:46:11.946555 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.946535    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bdpjh"
Jul 31 23:46:11.947307 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.947286    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 23:46:11.947473 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.947347    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 23:46:11.947473 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.947398    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 23:46:11.947945 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.947602    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 23:46:11.947945 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.947800    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:11.947945 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.947818    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 23:46:11.947945 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.947860    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:11.951253 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.948448    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jsrzq\""
Jul 31 23:46:11.951253 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.948581    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:11.951253 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.949520    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wl482\""
Jul 31 23:46:11.951253 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.949593    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 23:46:11.951253 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.949757    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 23:46:11.951253 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.951217    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-registration-dir\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:11.951755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.951401    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdmkz\" (UniqueName: \"kubernetes.io/projected/a66649bd-a671-4bee-af65-7930bfc49e55-kube-api-access-sdmkz\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:11.951755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.951472    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs\") pod \"network-metrics-daemon-5qcqv\" (UID: \"1fcbea3e-f254-440f-9281-bb84b6c5afaf\") " pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:11.951755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.951503    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpgvn\" (UniqueName: \"kubernetes.io/projected/1fcbea3e-f254-440f-9281-bb84b6c5afaf-kube-api-access-fpgvn\") pod \"network-metrics-daemon-5qcqv\" (UID: \"1fcbea3e-f254-440f-9281-bb84b6c5afaf\") " pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:11.951755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.951539    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-kubelet-config\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:11.951755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.951570    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-device-dir\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:11.951755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.951616    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a66649bd-a671-4bee-af65-7930bfc49e55-cnibin\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:11.951755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.951650    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a66649bd-a671-4bee-af65-7930bfc49e55-cni-binary-copy\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:11.951755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.951679    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a66649bd-a671-4bee-af65-7930bfc49e55-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:11.951755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.951713    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a66649bd-a671-4bee-af65-7930bfc49e55-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:11.951755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.951749    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbqms\" (UniqueName: \"kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms\") pod \"network-check-target-wr59z\" (UID: \"4c5d6736-af34-4032-9c3d-d7e95b856f36\") " pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:11.952258 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.951790    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-socket-dir\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:11.952258 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.951828    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:11.952258 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.951867    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a66649bd-a671-4bee-af65-7930bfc49e55-system-cni-dir\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:11.952258 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.952237    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 23:46:11.952564 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.952546    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kf2cg\""
Jul 31 23:46:11.952631 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.952583    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a66649bd-a671-4bee-af65-7930bfc49e55-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:11.952870 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.952625    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-sys-fs\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:11.952870 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.952783    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dxxt\" (UniqueName: \"kubernetes.io/projected/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-kube-api-access-6dxxt\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:11.952870 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.952830    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a66649bd-a671-4bee-af65-7930bfc49e55-os-release\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:11.953125 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.952867    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-dbus\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:11.953125 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.953031    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:11.953125 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.953075    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:11.954686 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.954662    2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 23:46:11.974485 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.974453    2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lrgsr"
Jul 31 23:46:11.982120 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:11.982096    2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lrgsr"
Jul 31 23:46:12.037795 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.037768    2546 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 23:46:12.053637 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.053613    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-sysctl-d\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.053774 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.053642    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-cnibin\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.053774 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.053660    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-run-k8s-cni-cncf-io\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.053774 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.053680    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-multus-daemon-config\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.053774 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.053712    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a66649bd-a671-4bee-af65-7930bfc49e55-os-release\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.053774 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.053738    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:12.054039 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.053825    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a66649bd-a671-4bee-af65-7930bfc49e55-os-release\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.054039 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.053851    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:12.054039 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.053869    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-systemd\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.054039 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.053898    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-run\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.054039 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.053926    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret podName:4eb3a0b2-c573-43c7-a8dd-a01c3545d78b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:12.553891587 +0000 UTC m=+3.013354404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret") pod "global-pull-secret-syncer-zmcw2" (UID: "4eb3a0b2-c573-43c7-a8dd-a01c3545d78b") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:12.054039 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054026    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-etc-kubernetes\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.054320 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054063    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpgvn\" (UniqueName: \"kubernetes.io/projected/1fcbea3e-f254-440f-9281-bb84b6c5afaf-kube-api-access-fpgvn\") pod \"network-metrics-daemon-5qcqv\" (UID: \"1fcbea3e-f254-440f-9281-bb84b6c5afaf\") " pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:12.054320 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054089    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5278364c-d649-48ba-8388-d277b82f21a7-host\") pod \"node-ca-hp4zj\" (UID: \"5278364c-d649-48ba-8388-d277b82f21a7\") " pod="openshift-image-registry/node-ca-hp4zj"
Jul 31 23:46:12.054320 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054113    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-sysconfig\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.054320 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054140    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-lib-modules\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.054320 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054164    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-run-ovn\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.054320 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054191    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-var-lib-cni-bin\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.054320 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054218    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-var-lib-cni-multus\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.054320 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054247    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbqms\" (UniqueName: \"kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms\") pod \"network-check-target-wr59z\" (UID: \"4c5d6736-af34-4032-9c3d-d7e95b856f36\") " pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:12.054320 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054299    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-modprobe-d\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.054741 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054348    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f883b5d2-c71f-40f7-bdbd-131e956f4cef-ovnkube-config\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.054741 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054402    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-sysctl-conf\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.054741 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054439    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-var-lib-kubelet\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.054741 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054481    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4b8\" (UniqueName: \"kubernetes.io/projected/0fe4a567-1748-4bd0-bc6f-244498c60b37-kube-api-access-kv4b8\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.054741 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054509    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-run-systemd\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.054741 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054536    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-tuned\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.054741 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054562    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsldt\" (UniqueName: \"kubernetes.io/projected/655fd51c-e55e-4fa9-8a11-8de5f622b670-kube-api-access-nsldt\") pod \"node-resolver-v58mz\" (UID: \"655fd51c-e55e-4fa9-8a11-8de5f622b670\") " pod="openshift-dns/node-resolver-v58mz"
Jul 31 23:46:12.054741 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054586    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-cni-binary-copy\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.054741 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054611    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-var-lib-kubelet\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.054741 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054638    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-device-dir\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:12.054741 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054674    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85746df1-2720-4d80-96a6-ed16dbee95a8-iptables-alerter-script\") pod \"iptables-alerter-l94lx\" (UID: \"85746df1-2720-4d80-96a6-ed16dbee95a8\") " pod="openshift-network-operator/iptables-alerter-l94lx"
Jul 31 23:46:12.054741 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054694    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a66649bd-a671-4bee-af65-7930bfc49e55-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.054741 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054720    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-socket-dir\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:12.054741 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054743    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5278364c-d649-48ba-8388-d277b82f21a7-serviceca\") pod \"node-ca-hp4zj\" (UID: \"5278364c-d649-48ba-8388-d277b82f21a7\") " pod="openshift-image-registry/node-ca-hp4zj"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054767    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-slash\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054770    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-device-dir\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054790    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-etc-openvswitch\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054827    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-node-log\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054858    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-run-ovn-kubernetes\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054885    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-multus-socket-dir-parent\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054910    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-kubelet\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054936    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-socket-dir\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054945    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6skzt\" (UniqueName: \"kubernetes.io/projected/f883b5d2-c71f-40f7-bdbd-131e956f4cef-kube-api-access-6skzt\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.054988    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-os-release\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055018    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f76wk\" (UniqueName: \"kubernetes.io/projected/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-kube-api-access-f76wk\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055050    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055117    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055130    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-registration-dir\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055156    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/655fd51c-e55e-4fa9-8a11-8de5f622b670-tmp-dir\") pod \"node-resolver-v58mz\" (UID: \"655fd51c-e55e-4fa9-8a11-8de5f622b670\") " pod="openshift-dns/node-resolver-v58mz"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055179    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-log-socket\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.055466 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055202    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-system-cni-dir\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055228    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t728j\" (UniqueName: \"kubernetes.io/projected/5278364c-d649-48ba-8388-d277b82f21a7-kube-api-access-t728j\") pod \"node-ca-hp4zj\" (UID: \"5278364c-d649-48ba-8388-d277b82f21a7\") " pod="openshift-image-registry/node-ca-hp4zj"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055239    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-registration-dir\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055254    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-var-lib-openvswitch\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055310    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055326    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a66649bd-a671-4bee-af65-7930bfc49e55-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055352    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f883b5d2-c71f-40f7-bdbd-131e956f4cef-ovn-node-metrics-cert\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055380    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8xsv\" (UniqueName: \"kubernetes.io/projected/85746df1-2720-4d80-96a6-ed16dbee95a8-kube-api-access-r8xsv\") pod \"iptables-alerter-l94lx\" (UID: \"85746df1-2720-4d80-96a6-ed16dbee95a8\") " pod="openshift-network-operator/iptables-alerter-l94lx"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055406    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-multus-cni-dir\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055436    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055461    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-run-openvswitch\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055487    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f883b5d2-c71f-40f7-bdbd-131e956f4cef-env-overrides\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055513    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f883b5d2-c71f-40f7-bdbd-131e956f4cef-ovnkube-script-lib\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055539    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-run-multus-certs\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055576    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055578    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-sys-fs\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:12.056427 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055617    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dxxt\" (UniqueName: \"kubernetes.io/projected/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-kube-api-access-6dxxt\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055648    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-sys-fs\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055687    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-host\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055753    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-dbus\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055776    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6385e683-4bac-448e-b72d-240f54800c50-agent-certs\") pod \"konnectivity-agent-bdpjh\" (UID: \"6385e683-4bac-448e-b72d-240f54800c50\") " pod="kube-system/konnectivity-agent-bdpjh"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055814    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fe4a567-1748-4bd0-bc6f-244498c60b37-tmp\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055844    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-run-netns\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055874    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-hostroot\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055944    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmkz\" (UniqueName: \"kubernetes.io/projected/a66649bd-a671-4bee-af65-7930bfc49e55-kube-api-access-sdmkz\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055978    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs\") pod \"network-metrics-daemon-5qcqv\" (UID: \"1fcbea3e-f254-440f-9281-bb84b6c5afaf\") " pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.055984    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-dbus\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056006    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-kubelet-config\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056035    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6385e683-4bac-448e-b72d-240f54800c50-konnectivity-ca\") pod \"konnectivity-agent-bdpjh\" (UID: \"6385e683-4bac-448e-b72d-240f54800c50\") " pod="kube-system/konnectivity-agent-bdpjh"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056062    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-kubernetes\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.056065    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056087    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-sys\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056127    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/655fd51c-e55e-4fa9-8a11-8de5f622b670-hosts-file\") pod \"node-resolver-v58mz\" (UID: \"655fd51c-e55e-4fa9-8a11-8de5f622b670\") " pod="openshift-dns/node-resolver-v58mz"
Jul 31 23:46:12.057207 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.056137    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs podName:1fcbea3e-f254-440f-9281-bb84b6c5afaf nodeName:}" failed. No retries permitted until 2026-07-31 23:46:12.556122227 +0000 UTC m=+3.015585020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs") pod "network-metrics-daemon-5qcqv" (UID: "1fcbea3e-f254-440f-9281-bb84b6c5afaf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056135    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-kubelet-config\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056182    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-systemd-units\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056209    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a66649bd-a671-4bee-af65-7930bfc49e55-cnibin\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056225    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a66649bd-a671-4bee-af65-7930bfc49e55-cni-binary-copy\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056250    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a66649bd-a671-4bee-af65-7930bfc49e55-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056322    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a66649bd-a671-4bee-af65-7930bfc49e55-cnibin\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056344    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-run-netns\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056367    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-cni-bin\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056390    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-cni-netd\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056406    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85746df1-2720-4d80-96a6-ed16dbee95a8-host-slash\") pod \"iptables-alerter-l94lx\" (UID: \"85746df1-2720-4d80-96a6-ed16dbee95a8\") " pod="openshift-network-operator/iptables-alerter-l94lx"
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056421    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-multus-conf-dir\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056443    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a66649bd-a671-4bee-af65-7930bfc49e55-system-cni-dir\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056466    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a66649bd-a671-4bee-af65-7930bfc49e55-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056619    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a66649bd-a671-4bee-af65-7930bfc49e55-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056686    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a66649bd-a671-4bee-af65-7930bfc49e55-system-cni-dir\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.058095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.056975    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a66649bd-a671-4bee-af65-7930bfc49e55-cni-binary-copy\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.058927 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.057169    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a66649bd-a671-4bee-af65-7930bfc49e55-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.061023 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.061001    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:12.061023 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.061024    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:12.061194 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.061036    2546 projected.go:196] Error preparing data for projected volume kube-api-access-rbqms for pod openshift-network-diagnostics/network-check-target-wr59z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:12.061194 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.061113    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms podName:4c5d6736-af34-4032-9c3d-d7e95b856f36 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:12.5610965 +0000 UTC m=+3.020559316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rbqms" (UniqueName: "kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms") pod "network-check-target-wr59z" (UID: "4c5d6736-af34-4032-9c3d-d7e95b856f36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:12.061298 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.061219    2546 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 23:46:12.065845 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.065821    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpgvn\" (UniqueName: \"kubernetes.io/projected/1fcbea3e-f254-440f-9281-bb84b6c5afaf-kube-api-access-fpgvn\") pod \"network-metrics-daemon-5qcqv\" (UID: \"1fcbea3e-f254-440f-9281-bb84b6c5afaf\") " pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:12.065955 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.065820    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dxxt\" (UniqueName: \"kubernetes.io/projected/3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a-kube-api-access-6dxxt\") pod \"aws-ebs-csi-driver-node-rjbqr\" (UID: \"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:12.069805 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.069785    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdmkz\" (UniqueName: \"kubernetes.io/projected/a66649bd-a671-4bee-af65-7930bfc49e55-kube-api-access-sdmkz\") pod \"multus-additional-cni-plugins-4q4d7\" (UID: \"a66649bd-a671-4bee-af65-7930bfc49e55\") " pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.074698 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.074536    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-138.ec2.internal" event={"ID":"d1004082fdc42266a4ca7100481dac3f","Type":"ContainerStarted","Data":"1925a2ef67a9ade2966ce80338ab22d3b2ec83f8e7e9413902094d128c739c23"}
Jul 31 23:46:12.077008 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.076983    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal" event={"ID":"e68ec58a1ffc2594df23e7ea19433036","Type":"ContainerStarted","Data":"9c6eedd26d22854fc88ee85fc52bec0f03537b7ae3908d40f1c8fc9dc6353a90"}
Jul 31 23:46:12.157670 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.157630    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-var-lib-kubelet\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.157915 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.157675    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4b8\" (UniqueName: \"kubernetes.io/projected/0fe4a567-1748-4bd0-bc6f-244498c60b37-kube-api-access-kv4b8\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.157915 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.157702    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-run-systemd\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.157915 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.157729    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-tuned\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.157915 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.157756    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsldt\" (UniqueName: \"kubernetes.io/projected/655fd51c-e55e-4fa9-8a11-8de5f622b670-kube-api-access-nsldt\") pod \"node-resolver-v58mz\" (UID: \"655fd51c-e55e-4fa9-8a11-8de5f622b670\") " pod="openshift-dns/node-resolver-v58mz"
Jul 31 23:46:12.157915 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.157763    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-var-lib-kubelet\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.157915 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.157781    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-cni-binary-copy\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.157915 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.157825    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-var-lib-kubelet\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.157915 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.157856    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85746df1-2720-4d80-96a6-ed16dbee95a8-iptables-alerter-script\") pod \"iptables-alerter-l94lx\" (UID: \"85746df1-2720-4d80-96a6-ed16dbee95a8\") " pod="openshift-network-operator/iptables-alerter-l94lx"
Jul 31 23:46:12.157915 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.157883    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5278364c-d649-48ba-8388-d277b82f21a7-serviceca\") pod \"node-ca-hp4zj\" (UID: \"5278364c-d649-48ba-8388-d277b82f21a7\") " pod="openshift-image-registry/node-ca-hp4zj"
Jul 31 23:46:12.157915 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.157905    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-slash\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.157957    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-etc-openvswitch\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.157984    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-node-log\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158007    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-run-ovn-kubernetes\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158058    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-run-ovn-kubernetes\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158059    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-run-systemd\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158081    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-multus-socket-dir-parent\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158108    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-kubelet\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158134    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6skzt\" (UniqueName: \"kubernetes.io/projected/f883b5d2-c71f-40f7-bdbd-131e956f4cef-kube-api-access-6skzt\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158177    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-os-release\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158201    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f76wk\" (UniqueName: \"kubernetes.io/projected/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-kube-api-access-f76wk\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158219    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/655fd51c-e55e-4fa9-8a11-8de5f622b670-tmp-dir\") pod \"node-resolver-v58mz\" (UID: \"655fd51c-e55e-4fa9-8a11-8de5f622b670\") " pod="openshift-dns/node-resolver-v58mz"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158236    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-log-socket\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158252    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-system-cni-dir\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158279    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t728j\" (UniqueName: \"kubernetes.io/projected/5278364c-d649-48ba-8388-d277b82f21a7-kube-api-access-t728j\") pod \"node-ca-hp4zj\" (UID: \"5278364c-d649-48ba-8388-d277b82f21a7\") " pod="openshift-image-registry/node-ca-hp4zj"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158296    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-var-lib-openvswitch\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158313    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.158501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158353    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f883b5d2-c71f-40f7-bdbd-131e956f4cef-ovn-node-metrics-cert\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158376    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8xsv\" (UniqueName: \"kubernetes.io/projected/85746df1-2720-4d80-96a6-ed16dbee95a8-kube-api-access-r8xsv\") pod \"iptables-alerter-l94lx\" (UID: \"85746df1-2720-4d80-96a6-ed16dbee95a8\") " pod="openshift-network-operator/iptables-alerter-l94lx"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158393    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-multus-cni-dir\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158411    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-run-openvswitch\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158426    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f883b5d2-c71f-40f7-bdbd-131e956f4cef-env-overrides\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158450    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f883b5d2-c71f-40f7-bdbd-131e956f4cef-ovnkube-script-lib\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158471    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-run-multus-certs\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158489    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-host\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158511    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6385e683-4bac-448e-b72d-240f54800c50-agent-certs\") pod \"konnectivity-agent-bdpjh\" (UID: \"6385e683-4bac-448e-b72d-240f54800c50\") " pod="kube-system/konnectivity-agent-bdpjh"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158533    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fe4a567-1748-4bd0-bc6f-244498c60b37-tmp\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158553    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-run-netns\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158569    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-hostroot\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158597    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6385e683-4bac-448e-b72d-240f54800c50-konnectivity-ca\") pod \"konnectivity-agent-bdpjh\" (UID: \"6385e683-4bac-448e-b72d-240f54800c50\") " pod="kube-system/konnectivity-agent-bdpjh"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158616    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-kubernetes\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158643    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-sys\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158658    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/655fd51c-e55e-4fa9-8a11-8de5f622b670-hosts-file\") pod \"node-resolver-v58mz\" (UID: \"655fd51c-e55e-4fa9-8a11-8de5f622b670\") " pod="openshift-dns/node-resolver-v58mz"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158656    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5278364c-d649-48ba-8388-d277b82f21a7-serviceca\") pod \"node-ca-hp4zj\" (UID: \"5278364c-d649-48ba-8388-d277b82f21a7\") " pod="openshift-image-registry/node-ca-hp4zj"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158665    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-cni-binary-copy\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.159406 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158729    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-systemd-units\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158722    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-var-lib-kubelet\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158776    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-slash\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158820    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-etc-openvswitch\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158863    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-node-log\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158673    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-systemd-units\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158908    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-run-netns\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158936    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-cni-bin\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158966    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-cni-netd\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.158993    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85746df1-2720-4d80-96a6-ed16dbee95a8-host-slash\") pod \"iptables-alerter-l94lx\" (UID: \"85746df1-2720-4d80-96a6-ed16dbee95a8\") " pod="openshift-network-operator/iptables-alerter-l94lx"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159019    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-multus-conf-dir\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159051    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-sysctl-d\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159077    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-cnibin\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159113    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-run-k8s-cni-cncf-io\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159138    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-multus-daemon-config\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159183    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-systemd\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159208    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-run\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159235    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-etc-kubernetes\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.160305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159262    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5278364c-d649-48ba-8388-d277b82f21a7-host\") pod \"node-ca-hp4zj\" (UID: \"5278364c-d649-48ba-8388-d277b82f21a7\") " pod="openshift-image-registry/node-ca-hp4zj"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159290    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-sysconfig\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159317    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-lib-modules\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159366    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-run-ovn\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159387    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-log-socket\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159387    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-multus-cni-dir\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159432    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-run-openvswitch\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159437    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-cni-bin\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159446    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-run-netns\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159473    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-cni-netd\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159503    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-system-cni-dir\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159511    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/655fd51c-e55e-4fa9-8a11-8de5f622b670-tmp-dir\") pod \"node-resolver-v58mz\" (UID: \"655fd51c-e55e-4fa9-8a11-8de5f622b670\") " pod="openshift-dns/node-resolver-v58mz"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159523    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85746df1-2720-4d80-96a6-ed16dbee95a8-host-slash\") pod \"iptables-alerter-l94lx\" (UID: \"85746df1-2720-4d80-96a6-ed16dbee95a8\") " pod="openshift-network-operator/iptables-alerter-l94lx"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159572    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159604    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-var-lib-openvswitch\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159660    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-multus-socket-dir-parent\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159704    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-host-kubelet\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159823    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85746df1-2720-4d80-96a6-ed16dbee95a8-iptables-alerter-script\") pod \"iptables-alerter-l94lx\" (UID: \"85746df1-2720-4d80-96a6-ed16dbee95a8\") " pod="openshift-network-operator/iptables-alerter-l94lx"
Jul 31 23:46:12.161231 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159907    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-os-release\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159982    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-host\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160027    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f883b5d2-c71f-40f7-bdbd-131e956f4cef-ovnkube-script-lib\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160077    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-run-netns\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160119    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-var-lib-cni-bin\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160132    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-hostroot\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.159394    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-var-lib-cni-bin\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160180    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-var-lib-cni-multus\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160224    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-modprobe-d\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160229    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-kubernetes\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160031    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-run-multus-certs\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160184    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-sys\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160251    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f883b5d2-c71f-40f7-bdbd-131e956f4cef-ovnkube-config\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160286    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-sysctl-conf\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160513    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-sysctl-conf\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160564    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-var-lib-cni-multus\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160571    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6385e683-4bac-448e-b72d-240f54800c50-konnectivity-ca\") pod \"konnectivity-agent-bdpjh\" (UID: \"6385e683-4bac-448e-b72d-240f54800c50\") " pod="kube-system/konnectivity-agent-bdpjh"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160597    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-multus-conf-dir\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.162247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160652    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/655fd51c-e55e-4fa9-8a11-8de5f622b670-hosts-file\") pod \"node-resolver-v58mz\" (UID: \"655fd51c-e55e-4fa9-8a11-8de5f622b670\") " pod="openshift-dns/node-resolver-v58mz"
Jul 31 23:46:12.163394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160675    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-modprobe-d\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.163394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160682    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-sysctl-d\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.163394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160704    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-etc-kubernetes\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.163394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160743    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-sysconfig\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.163394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160764    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-cnibin\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.163394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160799    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-lib-modules\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.163394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160805    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5278364c-d649-48ba-8388-d277b82f21a7-host\") pod \"node-ca-hp4zj\" (UID: \"5278364c-d649-48ba-8388-d277b82f21a7\") " pod="openshift-image-registry/node-ca-hp4zj"
Jul 31 23:46:12.163394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160842    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-host-run-k8s-cni-cncf-io\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.163394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160845    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f883b5d2-c71f-40f7-bdbd-131e956f4cef-run-ovn\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.163394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160903    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-run\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.163394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.160956    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-systemd\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.163394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.161397    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-multus-daemon-config\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.163394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.161959    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f883b5d2-c71f-40f7-bdbd-131e956f4cef-env-overrides\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.163394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.162408    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f883b5d2-c71f-40f7-bdbd-131e956f4cef-ovnkube-config\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.164456 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.164429    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6385e683-4bac-448e-b72d-240f54800c50-agent-certs\") pod \"konnectivity-agent-bdpjh\" (UID: \"6385e683-4bac-448e-b72d-240f54800c50\") " pod="kube-system/konnectivity-agent-bdpjh"
Jul 31 23:46:12.165292 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.165128    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fe4a567-1748-4bd0-bc6f-244498c60b37-tmp\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.165870 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.165846    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0fe4a567-1748-4bd0-bc6f-244498c60b37-etc-tuned\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.166111 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.166082    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f883b5d2-c71f-40f7-bdbd-131e956f4cef-ovn-node-metrics-cert\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.171777 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.171720    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8xsv\" (UniqueName: \"kubernetes.io/projected/85746df1-2720-4d80-96a6-ed16dbee95a8-kube-api-access-r8xsv\") pod \"iptables-alerter-l94lx\" (UID: \"85746df1-2720-4d80-96a6-ed16dbee95a8\") " pod="openshift-network-operator/iptables-alerter-l94lx"
Jul 31 23:46:12.174588 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.174565    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsldt\" (UniqueName: \"kubernetes.io/projected/655fd51c-e55e-4fa9-8a11-8de5f622b670-kube-api-access-nsldt\") pod \"node-resolver-v58mz\" (UID: \"655fd51c-e55e-4fa9-8a11-8de5f622b670\") " pod="openshift-dns/node-resolver-v58mz"
Jul 31 23:46:12.175536 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.175496    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4b8\" (UniqueName: \"kubernetes.io/projected/0fe4a567-1748-4bd0-bc6f-244498c60b37-kube-api-access-kv4b8\") pod \"tuned-9bmfk\" (UID: \"0fe4a567-1748-4bd0-bc6f-244498c60b37\") " pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.176397 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.176376    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f76wk\" (UniqueName: \"kubernetes.io/projected/45223f8f-c4dd-4b50-9c18-b182fc6e3fb7-kube-api-access-f76wk\") pod \"multus-hjsqg\" (UID: \"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7\") " pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.178147 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.178124    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t728j\" (UniqueName: \"kubernetes.io/projected/5278364c-d649-48ba-8388-d277b82f21a7-kube-api-access-t728j\") pod \"node-ca-hp4zj\" (UID: \"5278364c-d649-48ba-8388-d277b82f21a7\") " pod="openshift-image-registry/node-ca-hp4zj"
Jul 31 23:46:12.178565 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.178524    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6skzt\" (UniqueName: \"kubernetes.io/projected/f883b5d2-c71f-40f7-bdbd-131e956f4cef-kube-api-access-6skzt\") pod \"ovnkube-node-2qmq9\" (UID: \"f883b5d2-c71f-40f7-bdbd-131e956f4cef\") " pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.245418 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.245386    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4q4d7"
Jul 31 23:46:12.254161 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.254139    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr"
Jul 31 23:46:12.262570 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.262543    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9bmfk"
Jul 31 23:46:12.268804 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.268645    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hp4zj"
Jul 31 23:46:12.275605 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.275586    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v58mz"
Jul 31 23:46:12.283658 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.283636    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:12.291870 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.291725    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l94lx"
Jul 31 23:46:12.300040 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.299889    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bdpjh"
Jul 31 23:46:12.305902 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.305884    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hjsqg"
Jul 31 23:46:12.358988 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.358969    2546 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:12.427853 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.427779    2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 23:46:12.564314 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.564290    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs\") pod \"network-metrics-daemon-5qcqv\" (UID: \"1fcbea3e-f254-440f-9281-bb84b6c5afaf\") " pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:12.564429 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.564348    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:12.564474 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.564451    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:12.564510 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.564495    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:12.564542 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.564515    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs podName:1fcbea3e-f254-440f-9281-bb84b6c5afaf nodeName:}" failed. No retries permitted until 2026-07-31 23:46:13.564493823 +0000 UTC m=+4.023956619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs") pod "network-metrics-daemon-5qcqv" (UID: "1fcbea3e-f254-440f-9281-bb84b6c5afaf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:12.564542 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.564534    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret podName:4eb3a0b2-c573-43c7-a8dd-a01c3545d78b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:13.564522703 +0000 UTC m=+4.023985495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret") pod "global-pull-secret-syncer-zmcw2" (UID: "4eb3a0b2-c573-43c7-a8dd-a01c3545d78b") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:12.564621 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.564551    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbqms\" (UniqueName: \"kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms\") pod \"network-check-target-wr59z\" (UID: \"4c5d6736-af34-4032-9c3d-d7e95b856f36\") " pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:12.564655 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.564630    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:12.564655 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.564642    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:12.564655 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.564650    2546 projected.go:196] Error preparing data for projected volume kube-api-access-rbqms for pod openshift-network-diagnostics/network-check-target-wr59z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:12.564750 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:12.564681    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms podName:4c5d6736-af34-4032-9c3d-d7e95b856f36 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:13.564673118 +0000 UTC m=+4.024135910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rbqms" (UniqueName: "kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms") pod "network-check-target-wr59z" (UID: "4c5d6736-af34-4032-9c3d-d7e95b856f36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:12.881623 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:12.881591    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66649bd_a671_4bee_af65_7930bfc49e55.slice/crio-d28d788b3358eccb64ed1d52e20beadbf3828369d742c6291758f9734df17264 WatchSource:0}: Error finding container d28d788b3358eccb64ed1d52e20beadbf3828369d742c6291758f9734df17264: Status 404 returned error can't find the container with id d28d788b3358eccb64ed1d52e20beadbf3828369d742c6291758f9734df17264
Jul 31 23:46:12.883040 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:12.883015    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655fd51c_e55e_4fa9_8a11_8de5f622b670.slice/crio-7ffcb983aec8abfcc8870bf502db22141e9242f1113749265ccfe6859bac62cb WatchSource:0}: Error finding container 7ffcb983aec8abfcc8870bf502db22141e9242f1113749265ccfe6859bac62cb: Status 404 returned error can't find the container with id 7ffcb983aec8abfcc8870bf502db22141e9242f1113749265ccfe6859bac62cb
Jul 31 23:46:12.884185 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:12.884097    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe4a567_1748_4bd0_bc6f_244498c60b37.slice/crio-ea82ccb41dc7b4b244be801df1ddd2a4555a28a163b27b936518ec0dec28c007 WatchSource:0}: Error finding container ea82ccb41dc7b4b244be801df1ddd2a4555a28a163b27b936518ec0dec28c007: Status 404 returned error can't find the container with id ea82ccb41dc7b4b244be801df1ddd2a4555a28a163b27b936518ec0dec28c007
Jul 31 23:46:12.984169 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.984127    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 23:41:12 +0000 UTC" deadline="2028-02-29 17:40:54.206052426 +0000 UTC"
Jul 31 23:46:12.984169 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:12.984161    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13865h54m41.221896092s"
Jul 31 23:46:13.079687 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:13.079647    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9bmfk" event={"ID":"0fe4a567-1748-4bd0-bc6f-244498c60b37","Type":"ContainerStarted","Data":"ea82ccb41dc7b4b244be801df1ddd2a4555a28a163b27b936518ec0dec28c007"}
Jul 31 23:46:13.080962 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:13.080931    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v58mz" event={"ID":"655fd51c-e55e-4fa9-8a11-8de5f622b670","Type":"ContainerStarted","Data":"7ffcb983aec8abfcc8870bf502db22141e9242f1113749265ccfe6859bac62cb"}
Jul 31 23:46:13.082188 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:13.082162    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4q4d7" event={"ID":"a66649bd-a671-4bee-af65-7930bfc49e55","Type":"ContainerStarted","Data":"d28d788b3358eccb64ed1d52e20beadbf3828369d742c6291758f9734df17264"}
Jul 31 23:46:13.208201 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:13.208174    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5278364c_d649_48ba_8388_d277b82f21a7.slice/crio-de02aecab655dc7d42df751f55d72a2b7e73b6e116ec482c9ae91f5bf5fd876a WatchSource:0}: Error finding container de02aecab655dc7d42df751f55d72a2b7e73b6e116ec482c9ae91f5bf5fd876a: Status 404 returned error can't find the container with id de02aecab655dc7d42df751f55d72a2b7e73b6e116ec482c9ae91f5bf5fd876a
Jul 31 23:46:13.210292 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:13.210159    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85746df1_2720_4d80_96a6_ed16dbee95a8.slice/crio-40ceec07e5e182bdc45539cbe6f1a335c7ce5c3cf7b638d1a534d0ef68bc46dc WatchSource:0}: Error finding container 40ceec07e5e182bdc45539cbe6f1a335c7ce5c3cf7b638d1a534d0ef68bc46dc: Status 404 returned error can't find the container with id 40ceec07e5e182bdc45539cbe6f1a335c7ce5c3cf7b638d1a534d0ef68bc46dc
Jul 31 23:46:13.212860 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:13.212783    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6385e683_4bac_448e_b72d_240f54800c50.slice/crio-a4110b6dcbead9e715f1a30c1b1aee447c179171323924a01c03a63995e56d40 WatchSource:0}: Error finding container a4110b6dcbead9e715f1a30c1b1aee447c179171323924a01c03a63995e56d40: Status 404 returned error can't find the container with id a4110b6dcbead9e715f1a30c1b1aee447c179171323924a01c03a63995e56d40
Jul 31 23:46:13.214506 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:13.214367    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4ebe4d_6984_4dc1_b954_ac77bb4a5a4a.slice/crio-da5eaff54f0ebcfd18d3b0dbde8da9463558855329906fe01f3f34dbf5bb39e1 WatchSource:0}: Error finding container da5eaff54f0ebcfd18d3b0dbde8da9463558855329906fe01f3f34dbf5bb39e1: Status 404 returned error can't find the container with id da5eaff54f0ebcfd18d3b0dbde8da9463558855329906fe01f3f34dbf5bb39e1
Jul 31 23:46:13.215383 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:13.215278    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf883b5d2_c71f_40f7_bdbd_131e956f4cef.slice/crio-c6e1f7a536d8deed3e6e9f8c4c2bd8280e953789f99f177351ad3f225a104dd3 WatchSource:0}: Error finding container c6e1f7a536d8deed3e6e9f8c4c2bd8280e953789f99f177351ad3f225a104dd3: Status 404 returned error can't find the container with id c6e1f7a536d8deed3e6e9f8c4c2bd8280e953789f99f177351ad3f225a104dd3
Jul 31 23:46:13.216157 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:13.216120    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45223f8f_c4dd_4b50_9c18_b182fc6e3fb7.slice/crio-c8be1f704ef7f69c3679c1478ea15a2f27a43b7ab222a8beafac83ffc6d428e2 WatchSource:0}: Error finding container c8be1f704ef7f69c3679c1478ea15a2f27a43b7ab222a8beafac83ffc6d428e2: Status 404 returned error can't find the container with id c8be1f704ef7f69c3679c1478ea15a2f27a43b7ab222a8beafac83ffc6d428e2
Jul 31 23:46:13.574400 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:13.573741    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs\") pod \"network-metrics-daemon-5qcqv\" (UID: \"1fcbea3e-f254-440f-9281-bb84b6c5afaf\") " pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:13.574400 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:13.573996    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:13.574400 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:13.574033    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:13.574400 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:13.574085    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs podName:1fcbea3e-f254-440f-9281-bb84b6c5afaf nodeName:}" failed. No retries permitted until 2026-07-31 23:46:15.574061594 +0000 UTC m=+6.033524403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs") pod "network-metrics-daemon-5qcqv" (UID: "1fcbea3e-f254-440f-9281-bb84b6c5afaf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:13.574400 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:13.574116    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbqms\" (UniqueName: \"kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms\") pod \"network-check-target-wr59z\" (UID: \"4c5d6736-af34-4032-9c3d-d7e95b856f36\") " pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:13.574400 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:13.574144    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:13.574400 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:13.574196    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret podName:4eb3a0b2-c573-43c7-a8dd-a01c3545d78b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:15.574179638 +0000 UTC m=+6.033642438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret") pod "global-pull-secret-syncer-zmcw2" (UID: "4eb3a0b2-c573-43c7-a8dd-a01c3545d78b") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:13.574400 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:13.574260    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:13.574400 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:13.574293    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:13.574400 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:13.574307    2546 projected.go:196] Error preparing data for projected volume kube-api-access-rbqms for pod openshift-network-diagnostics/network-check-target-wr59z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:13.574400 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:13.574349    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms podName:4c5d6736-af34-4032-9c3d-d7e95b856f36 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:15.574335353 +0000 UTC m=+6.033798151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rbqms" (UniqueName: "kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms") pod "network-check-target-wr59z" (UID: "4c5d6736-af34-4032-9c3d-d7e95b856f36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:13.985420 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:13.985249    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 23:41:12 +0000 UTC" deadline="2028-01-31 06:38:14.522358714 +0000 UTC"
Jul 31 23:46:13.985420 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:13.985325    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13158h52m0.537039175s"
Jul 31 23:46:14.072901 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:14.072852    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:14.073076 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:14.073021    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:14.073476 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:14.073438    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:14.073588 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:14.073537    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:14.073667 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:14.073604    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:14.073741 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:14.073675    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:14.106347 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:14.101153    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-138.ec2.internal" event={"ID":"d1004082fdc42266a4ca7100481dac3f","Type":"ContainerStarted","Data":"b01c8a4479ba30e5c65c061e288c2e58db24141d2062d06b020f4a4dc812b2f8"}
Jul 31 23:46:14.116399 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:14.111516    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hjsqg" event={"ID":"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7","Type":"ContainerStarted","Data":"c8be1f704ef7f69c3679c1478ea15a2f27a43b7ab222a8beafac83ffc6d428e2"}
Jul 31 23:46:14.124667 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:14.124639    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9" event={"ID":"f883b5d2-c71f-40f7-bdbd-131e956f4cef","Type":"ContainerStarted","Data":"c6e1f7a536d8deed3e6e9f8c4c2bd8280e953789f99f177351ad3f225a104dd3"}
Jul 31 23:46:14.128855 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:14.128823    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr" event={"ID":"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a","Type":"ContainerStarted","Data":"da5eaff54f0ebcfd18d3b0dbde8da9463558855329906fe01f3f34dbf5bb39e1"}
Jul 31 23:46:14.142117 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:14.142039    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bdpjh" event={"ID":"6385e683-4bac-448e-b72d-240f54800c50","Type":"ContainerStarted","Data":"a4110b6dcbead9e715f1a30c1b1aee447c179171323924a01c03a63995e56d40"}
Jul 31 23:46:14.159409 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:14.159349    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l94lx" event={"ID":"85746df1-2720-4d80-96a6-ed16dbee95a8","Type":"ContainerStarted","Data":"40ceec07e5e182bdc45539cbe6f1a335c7ce5c3cf7b638d1a534d0ef68bc46dc"}
Jul 31 23:46:14.174731 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:14.174705    2546 generic.go:356] "Generic (PLEG): container finished" podID="e68ec58a1ffc2594df23e7ea19433036" containerID="29647613dc630fa9da16f8389e92b4be36ec4dbba4d6aca54b9f6cdc7dc67a61" exitCode=0
Jul 31 23:46:14.174828 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:14.174811    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal" event={"ID":"e68ec58a1ffc2594df23e7ea19433036","Type":"ContainerDied","Data":"29647613dc630fa9da16f8389e92b4be36ec4dbba4d6aca54b9f6cdc7dc67a61"}
Jul 31 23:46:14.192363 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:14.192302    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hp4zj" event={"ID":"5278364c-d649-48ba-8388-d277b82f21a7","Type":"ContainerStarted","Data":"de02aecab655dc7d42df751f55d72a2b7e73b6e116ec482c9ae91f5bf5fd876a"}
Jul 31 23:46:14.194065 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:14.194011    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-138.ec2.internal" podStartSLOduration=2.193991747 podStartE2EDuration="2.193991747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:14.116915307 +0000 UTC m=+4.576378125" watchObservedRunningTime="2026-07-31 23:46:14.193991747 +0000 UTC m=+4.653454566"
Jul 31 23:46:15.226518 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:15.226414    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal" event={"ID":"e68ec58a1ffc2594df23e7ea19433036","Type":"ContainerStarted","Data":"e3f65857d0cf80153b92a4f62b47541f340a328bece1090978fa12b6eb44c9cd"}
Jul 31 23:46:15.243519 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:15.243469    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-138.ec2.internal" podStartSLOduration=3.243448787 podStartE2EDuration="3.243448787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:15.243239211 +0000 UTC m=+5.702702055" watchObservedRunningTime="2026-07-31 23:46:15.243448787 +0000 UTC m=+5.702911605"
Jul 31 23:46:15.591215 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:15.591100    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:15.591215 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:15.591155    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbqms\" (UniqueName: \"kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms\") pod \"network-check-target-wr59z\" (UID: \"4c5d6736-af34-4032-9c3d-d7e95b856f36\") " pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:15.591215 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:15.591216    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs\") pod \"network-metrics-daemon-5qcqv\" (UID: \"1fcbea3e-f254-440f-9281-bb84b6c5afaf\") " pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:15.591523 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:15.591238    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:15.591523 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:15.591306    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:15.591523 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:15.591322    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret podName:4eb3a0b2-c573-43c7-a8dd-a01c3545d78b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:19.591302774 +0000 UTC m=+10.050765582 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret") pod "global-pull-secret-syncer-zmcw2" (UID: "4eb3a0b2-c573-43c7-a8dd-a01c3545d78b") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:15.591523 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:15.591344    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs podName:1fcbea3e-f254-440f-9281-bb84b6c5afaf nodeName:}" failed. No retries permitted until 2026-07-31 23:46:19.591333062 +0000 UTC m=+10.050795854 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs") pod "network-metrics-daemon-5qcqv" (UID: "1fcbea3e-f254-440f-9281-bb84b6c5afaf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:15.591523 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:15.591417    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:15.591523 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:15.591432    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:15.591523 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:15.591447    2546 projected.go:196] Error preparing data for projected volume kube-api-access-rbqms for pod openshift-network-diagnostics/network-check-target-wr59z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:15.591523 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:15.591482    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms podName:4c5d6736-af34-4032-9c3d-d7e95b856f36 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:19.591470841 +0000 UTC m=+10.050933655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rbqms" (UniqueName: "kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms") pod "network-check-target-wr59z" (UID: "4c5d6736-af34-4032-9c3d-d7e95b856f36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:16.075796 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:16.074375    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:16.075796 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:16.074521    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:16.075796 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:16.074583    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:16.075796 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:16.074652    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:16.075796 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:16.074694    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:16.075796 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:16.074766    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:18.070834 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:18.070802    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:18.071305 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:18.070932    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:18.071387 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:18.071352    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:18.071466 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:18.071443    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:18.071598 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:18.071582    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:18.071684 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:18.071665    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:19.625053 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:19.625016    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs\") pod \"network-metrics-daemon-5qcqv\" (UID: \"1fcbea3e-f254-440f-9281-bb84b6c5afaf\") " pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:19.625526 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:19.625083    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:19.625526 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:19.625118    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbqms\" (UniqueName: \"kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms\") pod \"network-check-target-wr59z\" (UID: \"4c5d6736-af34-4032-9c3d-d7e95b856f36\") " pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:19.625526 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:19.625278    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:19.625526 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:19.625301    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:19.625526 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:19.625314    2546 projected.go:196] Error preparing data for projected volume kube-api-access-rbqms for pod openshift-network-diagnostics/network-check-target-wr59z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:19.625526 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:19.625372    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms podName:4c5d6736-af34-4032-9c3d-d7e95b856f36 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:27.625352587 +0000 UTC m=+18.084815411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rbqms" (UniqueName: "kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms") pod "network-check-target-wr59z" (UID: "4c5d6736-af34-4032-9c3d-d7e95b856f36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:19.625832 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:19.625614    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:19.625832 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:19.625665    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret podName:4eb3a0b2-c573-43c7-a8dd-a01c3545d78b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:27.625652893 +0000 UTC m=+18.085115687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret") pod "global-pull-secret-syncer-zmcw2" (UID: "4eb3a0b2-c573-43c7-a8dd-a01c3545d78b") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:19.625832 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:19.625714    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:19.625832 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:19.625742    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs podName:1fcbea3e-f254-440f-9281-bb84b6c5afaf nodeName:}" failed. No retries permitted until 2026-07-31 23:46:27.625734244 +0000 UTC m=+18.085197034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs") pod "network-metrics-daemon-5qcqv" (UID: "1fcbea3e-f254-440f-9281-bb84b6c5afaf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:20.074210 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:20.073441    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:20.074210 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:20.073558    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:20.074210 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:20.073905    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:20.074210 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:20.074004    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:20.074210 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:20.074048    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:20.074210 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:20.074114    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:22.071221 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:22.070711    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:22.071221 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:22.070713    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:22.071221 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:22.070850    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:22.071221 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:22.070944    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:22.071221 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:22.070713    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:22.071221 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:22.071092    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:24.070299 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:24.070276    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:24.070642 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:24.070300    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:24.070642 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:24.070278    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:24.070642 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:24.070413    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:24.070642 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:24.070554    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:24.070788 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:24.070643    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:26.070906 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:26.070871    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:26.070906 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:26.070891    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:26.071436 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:26.070871    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:26.071436 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:26.071007    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:26.071436 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:26.071072    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:26.071436 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:26.071156    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:27.687817 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:27.687782    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs\") pod \"network-metrics-daemon-5qcqv\" (UID: \"1fcbea3e-f254-440f-9281-bb84b6c5afaf\") " pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:27.688316 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:27.687827    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:27.688316 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:27.687890    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbqms\" (UniqueName: \"kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms\") pod \"network-check-target-wr59z\" (UID: \"4c5d6736-af34-4032-9c3d-d7e95b856f36\") " pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:27.688316 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:27.687920    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:27.688316 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:27.687978    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs podName:1fcbea3e-f254-440f-9281-bb84b6c5afaf nodeName:}" failed. No retries permitted until 2026-07-31 23:46:43.687962252 +0000 UTC m=+34.147425043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs") pod "network-metrics-daemon-5qcqv" (UID: "1fcbea3e-f254-440f-9281-bb84b6c5afaf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:27.688316 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:27.688010    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:27.688316 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:27.688015    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:27.688316 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:27.688029    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:27.688316 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:27.688042    2546 projected.go:196] Error preparing data for projected volume kube-api-access-rbqms for pod openshift-network-diagnostics/network-check-target-wr59z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:27.688316 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:27.688067    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret podName:4eb3a0b2-c573-43c7-a8dd-a01c3545d78b nodeName:}" failed. No retries permitted until 2026-07-31 23:46:43.688052387 +0000 UTC m=+34.147515178 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret") pod "global-pull-secret-syncer-zmcw2" (UID: "4eb3a0b2-c573-43c7-a8dd-a01c3545d78b") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:27.688316 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:27.688088    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms podName:4c5d6736-af34-4032-9c3d-d7e95b856f36 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:43.688073364 +0000 UTC m=+34.147536177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rbqms" (UniqueName: "kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms") pod "network-check-target-wr59z" (UID: "4c5d6736-af34-4032-9c3d-d7e95b856f36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:28.070972 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:28.070892    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:28.071128 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:28.070899    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:28.071128 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:28.071043    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:28.071128 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:28.071091    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:28.071304 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:28.071216    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:28.071373 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:28.071309    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:30.070837 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:30.070802    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:30.071287 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:30.070903    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:30.071377 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:30.071299    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:30.071428 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:30.071387    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:30.071490 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:30.071449    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:30.071558 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:30.071540    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:32.070150 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:32.070128    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:32.070518 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:32.070128    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:32.070518 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:32.070133    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:32.070518 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:32.070265    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:32.070518 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:32.070366    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:32.070518 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:32.070423    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:32.261430 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:32.261211    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9bmfk" event={"ID":"0fe4a567-1748-4bd0-bc6f-244498c60b37","Type":"ContainerStarted","Data":"ae176f507ac3df54e47930013ba72179bea0dbad49c3a80a86c367d20288902f"}
Jul 31 23:46:32.263301 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:32.263220    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hp4zj" event={"ID":"5278364c-d649-48ba-8388-d277b82f21a7","Type":"ContainerStarted","Data":"9936646e679c988b60ea1919d325c3c3301f3a5678f679823c79bd0ed604f312"}
Jul 31 23:46:32.265517 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:32.265036    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4q4d7" event={"ID":"a66649bd-a671-4bee-af65-7930bfc49e55","Type":"ContainerStarted","Data":"5469d0844775c278def9d326e1e064a93e1f251c16134ce4b3411cd0ba8e61f4"}
Jul 31 23:46:32.267179 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:32.267130    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hjsqg" event={"ID":"45223f8f-c4dd-4b50-9c18-b182fc6e3fb7","Type":"ContainerStarted","Data":"da210836bec8b8470bb0751a703f46d709ca3e0100a22e71b14240b720fe70aa"}
Jul 31 23:46:32.274088 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:32.273987    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9" event={"ID":"f883b5d2-c71f-40f7-bdbd-131e956f4cef","Type":"ContainerStarted","Data":"ce7f1d2094ea146170e12428a99bf107f381e9539f732b24a2646a6d2054ab73"}
Jul 31 23:46:32.276735 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:32.276523    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bdpjh" event={"ID":"6385e683-4bac-448e-b72d-240f54800c50","Type":"ContainerStarted","Data":"706d4c271eb91cf74da4677369fea8d980c57e5cc8a1a7a60b68fa986e9c0709"}
Jul 31 23:46:32.288722 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:32.288666    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9bmfk" podStartSLOduration=3.132677145 podStartE2EDuration="22.288650837s" totalImagesPullingTime="19.155973692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:32.288384249 +0000 UTC m=+22.747847062" watchObservedRunningTime="2026-07-31 23:46:32.288650837 +0000 UTC m=+22.748113649"
Jul 31 23:46:32.335591 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:32.335292    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hp4zj" podStartSLOduration=3.505791767 podStartE2EDuration="22.335270359s" totalImagesPullingTime="18.829478592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:32.30869794 +0000 UTC m=+22.768160752" watchObservedRunningTime="2026-07-31 23:46:32.335270359 +0000 UTC m=+22.794733166"
Jul 31 23:46:32.335913 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:32.335876    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bdpjh" podStartSLOduration=11.619818217 podStartE2EDuration="22.335854117s" totalImagesPullingTime="10.7160359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:32.33437406 +0000 UTC m=+22.793836883" watchObservedRunningTime="2026-07-31 23:46:32.335854117 +0000 UTC m=+22.795316930"
Jul 31 23:46:32.356529 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:32.356402    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hjsqg" podStartSLOduration=3.511130755 podStartE2EDuration="22.356384704s" totalImagesPullingTime="18.845253949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:32.355900073 +0000 UTC m=+22.815362899" watchObservedRunningTime="2026-07-31 23:46:32.356384704 +0000 UTC m=+22.815847539"
Jul 31 23:46:33.279817 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:33.279784    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v58mz" event={"ID":"655fd51c-e55e-4fa9-8a11-8de5f622b670","Type":"ContainerStarted","Data":"d08d5e20fe0e870a4afa7b81f14ba8b14d2427f7972241832ef546cb26b46aa2"}
Jul 31 23:46:33.281230 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:33.281199    2546 generic.go:356] "Generic (PLEG): container finished" podID="a66649bd-a671-4bee-af65-7930bfc49e55" containerID="5469d0844775c278def9d326e1e064a93e1f251c16134ce4b3411cd0ba8e61f4" exitCode=0
Jul 31 23:46:33.281341 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:33.281278    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4q4d7" event={"ID":"a66649bd-a671-4bee-af65-7930bfc49e55","Type":"ContainerDied","Data":"5469d0844775c278def9d326e1e064a93e1f251c16134ce4b3411cd0ba8e61f4"}
Jul 31 23:46:33.284076 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:33.284050    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9" event={"ID":"f883b5d2-c71f-40f7-bdbd-131e956f4cef","Type":"ContainerStarted","Data":"a9d080660b5669c12ff454cf2268a2a03b6cd32b38c81c63821f9fd7514ed5ab"}
Jul 31 23:46:33.284159 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:33.284115    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9" event={"ID":"f883b5d2-c71f-40f7-bdbd-131e956f4cef","Type":"ContainerStarted","Data":"a38126644f24bfb265e4dae4fec14a094ca884fb638d7b12a2e235088ab927a2"}
Jul 31 23:46:33.284159 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:33.284153    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9" event={"ID":"f883b5d2-c71f-40f7-bdbd-131e956f4cef","Type":"ContainerStarted","Data":"8a178b54f6c9da74b52ad17a86eec56396caa680b32e303a7b78068068ec5041"}
Jul 31 23:46:33.284239 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:33.284182    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9" event={"ID":"f883b5d2-c71f-40f7-bdbd-131e956f4cef","Type":"ContainerStarted","Data":"dd2e86568d38991c40cbefe2d53ad8536d2a526fbae7758c6bb1e304dde6fa92"}
Jul 31 23:46:33.284239 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:33.284225    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9" event={"ID":"f883b5d2-c71f-40f7-bdbd-131e956f4cef","Type":"ContainerStarted","Data":"fc41da4b04286dea381ff82d2c11066ef40739641cd71f533fefc203d35e70df"}
Jul 31 23:46:33.285326 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:33.285295    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr" event={"ID":"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a","Type":"ContainerStarted","Data":"5ae2729f2fdf279100931bdc691e59b41ccfadae34148d2ffd3be72afd901522"}
Jul 31 23:46:33.297779 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:33.297743    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v58mz" podStartSLOduration=4.142957031 podStartE2EDuration="23.297729028s" totalImagesPullingTime="19.154771997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:33.297499758 +0000 UTC m=+23.756962572" watchObservedRunningTime="2026-07-31 23:46:33.297729028 +0000 UTC m=+23.757191840"
Jul 31 23:46:33.521189 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:33.521161    2546 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 23:46:33.996068 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:33.995968    2546 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T23:46:33.521182917Z","UUID":"efa261fc-0640-4279-ba87-9ea3676eb369","Handler":null,"Name":"","Endpoint":""}
Jul 31 23:46:33.997505 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:33.997482    2546 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 23:46:33.997640 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:33.997541    2546 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 23:46:34.071138 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:34.071106    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:34.071138 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:34.071121    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:34.071138 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:34.071106    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:34.071434 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:34.071233    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:34.071434 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:34.071311    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:34.071434 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:34.071387    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:34.289875 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:34.289799    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l94lx" event={"ID":"85746df1-2720-4d80-96a6-ed16dbee95a8","Type":"ContainerStarted","Data":"a72c76b6d97d1e45737b53e40fb2060b43c2a5a9aba9dd3e0bdb068d847e1134"}
Jul 31 23:46:34.292246 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:34.292178    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr" event={"ID":"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a","Type":"ContainerStarted","Data":"824a74d7cec0acef8248465c4adfaa433ee478a594e16fa4eea7e6ffaa5abdf6"}
Jul 31 23:46:34.306031 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:34.305980    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l94lx" podStartSLOduration=5.478602773 podStartE2EDuration="24.30596246s" totalImagesPullingTime="18.827359687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:34.305267889 +0000 UTC m=+24.764730697" watchObservedRunningTime="2026-07-31 23:46:34.30596246 +0000 UTC m=+24.765425275"
Jul 31 23:46:35.298698 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:35.298613    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9" event={"ID":"f883b5d2-c71f-40f7-bdbd-131e956f4cef","Type":"ContainerStarted","Data":"c7154794c8ef07010f1a3d86a39a6cb893f0939d0ae7d501e620ea98d164fccd"}
Jul 31 23:46:35.300789 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:35.300762    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr" event={"ID":"3c4ebe4d-6984-4dc1-b954-ac77bb4a5a4a","Type":"ContainerStarted","Data":"06abd6d3fb786fd32bfaeae9b11e26ebd0052c3167155a56a7b84550ae22a56a"}
Jul 31 23:46:35.318898 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:35.318851    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjbqr" podStartSLOduration=4.302979676 podStartE2EDuration="25.318837906s" totalImagesPullingTime="21.01585823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:35.318787554 +0000 UTC m=+25.778250369" watchObservedRunningTime="2026-07-31 23:46:35.318837906 +0000 UTC m=+25.778300719"
Jul 31 23:46:36.070582 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:36.070358    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:36.070762 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:36.070359    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:36.070762 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:36.070687    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:36.070762 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:36.070367    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:36.070918 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:36.070757    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:36.070918 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:36.070813    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:36.895865 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:36.895824    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bdpjh"
Jul 31 23:46:36.896482 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:36.896463    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bdpjh"
Jul 31 23:46:37.304658 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:37.304628    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bdpjh"
Jul 31 23:46:37.305050 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:37.305031    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bdpjh"
Jul 31 23:46:38.070488 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:38.070460    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:38.071122 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:38.070462    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:38.071122 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:38.070561    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:38.071122 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:38.070462    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:38.071122 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:38.070651    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:38.071122 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:38.070914    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:38.307591 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:38.307560    2546 generic.go:356] "Generic (PLEG): container finished" podID="a66649bd-a671-4bee-af65-7930bfc49e55" containerID="990192f129e4ad7282dc5145f2d1ba51c3c3b779bd4d90a01d86901fae3438bc" exitCode=0
Jul 31 23:46:38.307783 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:38.307653    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4q4d7" event={"ID":"a66649bd-a671-4bee-af65-7930bfc49e55","Type":"ContainerDied","Data":"990192f129e4ad7282dc5145f2d1ba51c3c3b779bd4d90a01d86901fae3438bc"}
Jul 31 23:46:38.311314 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:38.311286    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9" event={"ID":"f883b5d2-c71f-40f7-bdbd-131e956f4cef","Type":"ContainerStarted","Data":"77f28cdc026349766616c87484239096737e1d99a45165d0ed1660044b50b7a2"}
Jul 31 23:46:38.311691 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:38.311588    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:38.311691 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:38.311663    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:38.311796 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:38.311740    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:38.326291 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:38.326208    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:38.326369 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:38.326310    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:46:38.363075 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:38.363037    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9" podStartSLOduration=9.439945487 podStartE2EDuration="28.36302448s" totalImagesPullingTime="18.923078993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:38.362684948 +0000 UTC m=+28.822147760" watchObservedRunningTime="2026-07-31 23:46:38.36302448 +0000 UTC m=+28.822487293"
Jul 31 23:46:39.322094 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:39.322054    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zmcw2"]
Jul 31 23:46:39.322483 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:39.322195    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:39.322483 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:39.322331    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:39.323000 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:39.322979    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5qcqv"]
Jul 31 23:46:39.323096 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:39.323067    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:39.323271 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:39.323145    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:39.334763 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:39.334735    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wr59z"]
Jul 31 23:46:39.334867 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:39.334823    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:39.334984 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:39.334917    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:40.320292 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:40.320236    2546 generic.go:356] "Generic (PLEG): container finished" podID="a66649bd-a671-4bee-af65-7930bfc49e55" containerID="ad5c24c9431c3ab4a30b7eb10013bcf4ebe283a11a15d1a97aa38e9acc8e1740" exitCode=0
Jul 31 23:46:40.320485 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:40.320314    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4q4d7" event={"ID":"a66649bd-a671-4bee-af65-7930bfc49e55","Type":"ContainerDied","Data":"ad5c24c9431c3ab4a30b7eb10013bcf4ebe283a11a15d1a97aa38e9acc8e1740"}
Jul 31 23:46:41.070684 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:41.070649    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:41.071103 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:41.070736    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:41.071103 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:41.070744    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:41.071103 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:41.070799    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:41.071103 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:41.070918    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:41.071103 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:41.071020    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:42.128225 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:42.128050    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:42.211378 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:42.211347    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:42.326652 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:42.326620    2546 generic.go:356] "Generic (PLEG): container finished" podID="a66649bd-a671-4bee-af65-7930bfc49e55" containerID="39f7bd116f6e99caf50d4b4aff502e0fb14f2e366160589c3af64248698378cf" exitCode=0
Jul 31 23:46:42.326799 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:42.326676    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4q4d7" event={"ID":"a66649bd-a671-4bee-af65-7930bfc49e55","Type":"ContainerDied","Data":"39f7bd116f6e99caf50d4b4aff502e0fb14f2e366160589c3af64248698378cf"}
Jul 31 23:46:43.070343 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:43.070316    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:43.070521 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:43.070316    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:43.070521 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:43.070445    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qcqv" podUID="1fcbea3e-f254-440f-9281-bb84b6c5afaf"
Jul 31 23:46:43.070521 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:43.070316    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:43.070521 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:43.070512    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wr59z" podUID="4c5d6736-af34-4032-9c3d-d7e95b856f36"
Jul 31 23:46:43.070717 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:43.070563    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmcw2" podUID="4eb3a0b2-c573-43c7-a8dd-a01c3545d78b"
Jul 31 23:46:43.711428 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:43.711403    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbqms\" (UniqueName: \"kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms\") pod \"network-check-target-wr59z\" (UID: \"4c5d6736-af34-4032-9c3d-d7e95b856f36\") " pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:43.711987 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:43.711451    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs\") pod \"network-metrics-daemon-5qcqv\" (UID: \"1fcbea3e-f254-440f-9281-bb84b6c5afaf\") " pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:43.711987 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:43.711499    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:43.711987 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:43.711588    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 23:46:43.711987 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:43.711619    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 23:46:43.711987 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:43.711634    2546 projected.go:196] Error preparing data for projected volume kube-api-access-rbqms for pod openshift-network-diagnostics/network-check-target-wr59z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:43.711987 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:43.711593    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:43.711987 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:43.711693    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms podName:4c5d6736-af34-4032-9c3d-d7e95b856f36 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:15.711672532 +0000 UTC m=+66.171135325 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rbqms" (UniqueName: "kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms") pod "network-check-target-wr59z" (UID: "4c5d6736-af34-4032-9c3d-d7e95b856f36") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 23:46:43.711987 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:43.711732    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret podName:4eb3a0b2-c573-43c7-a8dd-a01c3545d78b nodeName:}" failed. No retries permitted until 2026-07-31 23:47:15.711716026 +0000 UTC m=+66.171178822 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret") pod "global-pull-secret-syncer-zmcw2" (UID: "4eb3a0b2-c573-43c7-a8dd-a01c3545d78b") : object "kube-system"/"original-pull-secret" not registered
Jul 31 23:46:43.711987 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:43.711593    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:43.711987 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:43.711766    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs podName:1fcbea3e-f254-440f-9281-bb84b6c5afaf nodeName:}" failed. No retries permitted until 2026-07-31 23:47:15.711758404 +0000 UTC m=+66.171221195 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs") pod "network-metrics-daemon-5qcqv" (UID: "1fcbea3e-f254-440f-9281-bb84b6c5afaf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 23:46:43.883271 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:43.883220    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:43.955168 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:43.955023    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:44.869697 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.868867    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-138.ec2.internal" event="NodeReady"
Jul 31 23:46:44.869697 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.869024    2546 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 23:46:44.921489 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.921459    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7m7vf"]
Jul 31 23:46:44.923367 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.923340    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7m7vf"
Jul 31 23:46:44.928810 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.928778    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 23:46:44.929179 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.929019    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 23:46:44.929179 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.929065    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 23:46:44.929478 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.929455    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-btr74\""
Jul 31 23:46:44.929744 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.929722    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jw8jw"]
Jul 31 23:46:44.931238 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.931219    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:44.934888 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.934864    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7m7vf"]
Jul 31 23:46:44.935125 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.935106    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 23:46:44.935125 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.935117    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r2mdc\""
Jul 31 23:46:44.935437 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.935296    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 23:46:44.948842 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.948810    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jw8jw"]
Jul 31 23:46:44.954752 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:44.954733    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:45.022379 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.022348    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:45.022544 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.022383    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-tmp-dir\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:45.022544 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.022417    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9mzz\" (UniqueName: \"kubernetes.io/projected/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-kube-api-access-t9mzz\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:45.022544 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.022461    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28fnc\" (UniqueName: \"kubernetes.io/projected/447bf0cf-4705-4db9-a4ba-cc6d257d3170-kube-api-access-28fnc\") pod \"ingress-canary-7m7vf\" (UID: \"447bf0cf-4705-4db9-a4ba-cc6d257d3170\") " pod="openshift-ingress-canary/ingress-canary-7m7vf"
Jul 31 23:46:45.022544 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.022511    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-config-volume\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:45.022544 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.022545    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/447bf0cf-4705-4db9-a4ba-cc6d257d3170-cert\") pod \"ingress-canary-7m7vf\" (UID: \"447bf0cf-4705-4db9-a4ba-cc6d257d3170\") " pod="openshift-ingress-canary/ingress-canary-7m7vf"
Jul 31 23:46:45.024150 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.024124    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:45.070989 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.070964    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:46:45.070989 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.070979    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:46:45.071186 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.070976    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:46:45.083876 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.083667    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 23:46:45.083876 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.083701    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 23:46:45.083876 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.083718    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zhtdr\""
Jul 31 23:46:45.083876 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.083674    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n5wdr\""
Jul 31 23:46:45.084495 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.084453    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 23:46:45.084607 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.084539    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 23:46:45.123185 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.123105    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/447bf0cf-4705-4db9-a4ba-cc6d257d3170-cert\") pod \"ingress-canary-7m7vf\" (UID: \"447bf0cf-4705-4db9-a4ba-cc6d257d3170\") " pod="openshift-ingress-canary/ingress-canary-7m7vf"
Jul 31 23:46:45.123185 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.123167    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:45.123387 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.123194    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-tmp-dir\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:45.123387 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.123224    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mzz\" (UniqueName: \"kubernetes.io/projected/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-kube-api-access-t9mzz\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:45.123387 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.123277    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28fnc\" (UniqueName: \"kubernetes.io/projected/447bf0cf-4705-4db9-a4ba-cc6d257d3170-kube-api-access-28fnc\") pod \"ingress-canary-7m7vf\" (UID: \"447bf0cf-4705-4db9-a4ba-cc6d257d3170\") " pod="openshift-ingress-canary/ingress-canary-7m7vf"
Jul 31 23:46:45.123387 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:45.123328    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:45.123572 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:45.123398    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls podName:fec5df48-6c8a-49ff-bb01-43d1d87ca8fd nodeName:}" failed. No retries permitted until 2026-07-31 23:46:45.623375507 +0000 UTC m=+36.082838301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls") pod "dns-default-jw8jw" (UID: "fec5df48-6c8a-49ff-bb01-43d1d87ca8fd") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:45.123572 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.123557    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-tmp-dir\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:45.123674 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.123330    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-config-volume\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:45.123975 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.123957    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-config-volume\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:45.127634 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.127613    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/447bf0cf-4705-4db9-a4ba-cc6d257d3170-cert\") pod \"ingress-canary-7m7vf\" (UID: \"447bf0cf-4705-4db9-a4ba-cc6d257d3170\") " pod="openshift-ingress-canary/ingress-canary-7m7vf"
Jul 31 23:46:45.132498 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.132468    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9mzz\" (UniqueName: \"kubernetes.io/projected/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-kube-api-access-t9mzz\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:45.132602 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.132579    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28fnc\" (UniqueName: \"kubernetes.io/projected/447bf0cf-4705-4db9-a4ba-cc6d257d3170-kube-api-access-28fnc\") pod \"ingress-canary-7m7vf\" (UID: \"447bf0cf-4705-4db9-a4ba-cc6d257d3170\") " pod="openshift-ingress-canary/ingress-canary-7m7vf"
Jul 31 23:46:45.236091 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.236058    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7m7vf"
Jul 31 23:46:45.627301 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:45.627244    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:45.627500 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:45.627423    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:45.627565 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:45.627512    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls podName:fec5df48-6c8a-49ff-bb01-43d1d87ca8fd nodeName:}" failed. No retries permitted until 2026-07-31 23:46:46.627488893 +0000 UTC m=+37.086951687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls") pod "dns-default-jw8jw" (UID: "fec5df48-6c8a-49ff-bb01-43d1d87ca8fd") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:46.569842 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:46.569720    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7m7vf"]
Jul 31 23:46:46.573138 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:46.573110    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447bf0cf_4705_4db9_a4ba_cc6d257d3170.slice/crio-cfd115465d168cb5e0c4e94187684d4208aa61b8f77b27f61b663d54632b9dd8 WatchSource:0}: Error finding container cfd115465d168cb5e0c4e94187684d4208aa61b8f77b27f61b663d54632b9dd8: Status 404 returned error can't find the container with id cfd115465d168cb5e0c4e94187684d4208aa61b8f77b27f61b663d54632b9dd8
Jul 31 23:46:46.634171 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:46.634140    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:46.634350 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:46.634320    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:46.634394 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:46.634382    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls podName:fec5df48-6c8a-49ff-bb01-43d1d87ca8fd nodeName:}" failed. No retries permitted until 2026-07-31 23:46:48.63436618 +0000 UTC m=+39.093828972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls") pod "dns-default-jw8jw" (UID: "fec5df48-6c8a-49ff-bb01-43d1d87ca8fd") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:47.342485 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.342455    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s"]
Jul 31 23:46:47.344485 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.344461    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s"
Jul 31 23:46:47.345814 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.345658    2546 generic.go:356] "Generic (PLEG): container finished" podID="a66649bd-a671-4bee-af65-7930bfc49e55" containerID="f35d9f72adc3842e0134226a01eae34f984e899ce4ed39bf77bc7e3aa584861c" exitCode=0
Jul 31 23:46:47.345814 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.345767    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4q4d7" event={"ID":"a66649bd-a671-4bee-af65-7930bfc49e55","Type":"ContainerDied","Data":"f35d9f72adc3842e0134226a01eae34f984e899ce4ed39bf77bc7e3aa584861c"}
Jul 31 23:46:47.347211 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.347188    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7m7vf" event={"ID":"447bf0cf-4705-4db9-a4ba-cc6d257d3170","Type":"ContainerStarted","Data":"cfd115465d168cb5e0c4e94187684d4208aa61b8f77b27f61b663d54632b9dd8"}
Jul 31 23:46:47.348214 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.348190    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 31 23:46:47.348310 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.348300    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:47.348396 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.348199    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:47.348514 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.348500    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nm9d4\""
Jul 31 23:46:47.348658 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.348647    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 31 23:46:47.356507 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.356477    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s"]
Jul 31 23:46:47.430745 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.430699    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-f44c6f5f8-5hkk9"]
Jul 31 23:46:47.433338 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.433316    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-ktsf2"]
Jul 31 23:46:47.433446 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.433436    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:47.435183 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.435166    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-ktsf2"
Jul 31 23:46:47.436337 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.436318    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 31 23:46:47.436561 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.436547    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2cnf6\""
Jul 31 23:46:47.436656 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.436635    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 31 23:46:47.436747 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.436706    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 31 23:46:47.436813 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.436755    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 31 23:46:47.436813 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.436777    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 31 23:46:47.437286 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.437271    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6rn62\""
Jul 31 23:46:47.437796 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.437778    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 31 23:46:47.441645 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.441469    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-default-certificate\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:47.441645 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.441512    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-stats-auth\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:47.441645 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.441539    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fp8b\" (UniqueName: \"kubernetes.io/projected/e005be7e-fed7-468e-9a7d-601db46f362f-kube-api-access-4fp8b\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:47.441645 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.441569    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:47.441645 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.441607    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e582be6-7c99-42ca-ad13-a97315764e09-serving-cert\") pod \"service-ca-operator-784f97bcf6-sbj9s\" (UID: \"4e582be6-7c99-42ca-ad13-a97315764e09\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s"
Jul 31 23:46:47.441980 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.441674    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:47.442034 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.441983    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxhzd\" (UniqueName: \"kubernetes.io/projected/4e582be6-7c99-42ca-ad13-a97315764e09-kube-api-access-rxhzd\") pod \"service-ca-operator-784f97bcf6-sbj9s\" (UID: \"4e582be6-7c99-42ca-ad13-a97315764e09\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s"
Jul 31 23:46:47.442090 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.442036    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e582be6-7c99-42ca-ad13-a97315764e09-config\") pod \"service-ca-operator-784f97bcf6-sbj9s\" (UID: \"4e582be6-7c99-42ca-ad13-a97315764e09\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s"
Jul 31 23:46:47.442090 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.442076    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjv9q\" (UniqueName: \"kubernetes.io/projected/8c85d11a-d782-4c52-9126-d9b563ed453f-kube-api-access-sjv9q\") pod \"network-check-source-65bdbbff45-ktsf2\" (UID: \"8c85d11a-d782-4c52-9126-d9b563ed453f\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-ktsf2"
Jul 31 23:46:47.453378 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.453328    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-ktsf2"]
Jul 31 23:46:47.459595 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.459571    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f44c6f5f8-5hkk9"]
Jul 31 23:46:47.534867 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.534838    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"]
Jul 31 23:46:47.536800 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.536781    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"]
Jul 31 23:46:47.536933 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.536915    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:46:47.538695 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.538676    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"
Jul 31 23:46:47.539480 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.539322    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 31 23:46:47.540246 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.540108    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 23:46:47.540246 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.540134    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-f464j\""
Jul 31 23:46:47.540413 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.540272    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 23:46:47.541165 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.541145    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f6b94dfb9-cngmx"]
Jul 31 23:46:47.541357 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.541340    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 31 23:46:47.542968 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.542910    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.543073 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.543011    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-default-certificate\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:47.543073 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.543041    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-stats-auth\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:47.543073 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.543066    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fp8b\" (UniqueName: \"kubernetes.io/projected/e005be7e-fed7-468e-9a7d-601db46f362f-kube-api-access-4fp8b\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:47.543338 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.543093    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:47.543338 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.543290    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e582be6-7c99-42ca-ad13-a97315764e09-serving-cert\") pod \"service-ca-operator-784f97bcf6-sbj9s\" (UID: \"4e582be6-7c99-42ca-ad13-a97315764e09\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s"
Jul 31 23:46:47.543338 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.543324    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:47.543563 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.543349    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxhzd\" (UniqueName: \"kubernetes.io/projected/4e582be6-7c99-42ca-ad13-a97315764e09-kube-api-access-rxhzd\") pod \"service-ca-operator-784f97bcf6-sbj9s\" (UID: \"4e582be6-7c99-42ca-ad13-a97315764e09\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s"
Jul 31 23:46:47.543563 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.543380    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e582be6-7c99-42ca-ad13-a97315764e09-config\") pod \"service-ca-operator-784f97bcf6-sbj9s\" (UID: \"4e582be6-7c99-42ca-ad13-a97315764e09\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s"
Jul 31 23:46:47.543563 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.543417    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjv9q\" (UniqueName: \"kubernetes.io/projected/8c85d11a-d782-4c52-9126-d9b563ed453f-kube-api-access-sjv9q\") pod \"network-check-source-65bdbbff45-ktsf2\" (UID: \"8c85d11a-d782-4c52-9126-d9b563ed453f\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-ktsf2"
Jul 31 23:46:47.543786 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:47.543764    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle podName:e005be7e-fed7-468e-9a7d-601db46f362f nodeName:}" failed. No retries permitted until 2026-07-31 23:46:48.043745796 +0000 UTC m=+38.503208607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle") pod "router-default-f44c6f5f8-5hkk9" (UID: "e005be7e-fed7-468e-9a7d-601db46f362f") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:47.544122 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.544099    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e582be6-7c99-42ca-ad13-a97315764e09-config\") pod \"service-ca-operator-784f97bcf6-sbj9s\" (UID: \"4e582be6-7c99-42ca-ad13-a97315764e09\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s"
Jul 31 23:46:47.544213 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.544164    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 31 23:46:47.544213 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.544195    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xzjn2\""
Jul 31 23:46:47.544213 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.544204    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 31 23:46:47.544492 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:47.544476    2546 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:47.544558 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.544481    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 31 23:46:47.544558 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:47.544528    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs podName:e005be7e-fed7-468e-9a7d-601db46f362f nodeName:}" failed. No retries permitted until 2026-07-31 23:46:48.044511905 +0000 UTC m=+38.503974712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs") pod "router-default-f44c6f5f8-5hkk9" (UID: "e005be7e-fed7-468e-9a7d-601db46f362f") : secret "router-metrics-certs-default" not found
Jul 31 23:46:47.547114 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.547094    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-stats-auth\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:47.547203 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.547167    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-default-certificate\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:47.547314 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.547295    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e582be6-7c99-42ca-ad13-a97315764e09-serving-cert\") pod \"service-ca-operator-784f97bcf6-sbj9s\" (UID: \"4e582be6-7c99-42ca-ad13-a97315764e09\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s"
Jul 31 23:46:47.548328 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.548308    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 31 23:46:47.548418 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.548351    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 31 23:46:47.548418 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.548396    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mbvf9\""
Jul 31 23:46:47.549826 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.549796    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 31 23:46:47.552396 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.552371    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"]
Jul 31 23:46:47.552769 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.552752    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"]
Jul 31 23:46:47.560518 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.560051    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f6b94dfb9-cngmx"]
Jul 31 23:46:47.562353 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.562333    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 31 23:46:47.567341 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.567319    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxhzd\" (UniqueName: \"kubernetes.io/projected/4e582be6-7c99-42ca-ad13-a97315764e09-kube-api-access-rxhzd\") pod \"service-ca-operator-784f97bcf6-sbj9s\" (UID: \"4e582be6-7c99-42ca-ad13-a97315764e09\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s"
Jul 31 23:46:47.567588 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.567570    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjv9q\" (UniqueName: \"kubernetes.io/projected/8c85d11a-d782-4c52-9126-d9b563ed453f-kube-api-access-sjv9q\") pod \"network-check-source-65bdbbff45-ktsf2\" (UID: \"8c85d11a-d782-4c52-9126-d9b563ed453f\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-ktsf2"
Jul 31 23:46:47.573182 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.573160    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fp8b\" (UniqueName: \"kubernetes.io/projected/e005be7e-fed7-468e-9a7d-601db46f362f-kube-api-access-4fp8b\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:47.605616 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.605555    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:47.644696 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.644664    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-bound-sa-token\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.644812 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.644711    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkgw4\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-kube-api-access-xkgw4\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.644812 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.644766    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59605a10-07f0-4484-bdda-7779bc9c8316-trusted-ca\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.644906 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.644838    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-c8dsw\" (UID: \"81b34b10-cd40-4782-b31e-086b1aa66312\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:46:47.644906 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.644892    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rc6z\" (UniqueName: \"kubernetes.io/projected/81b34b10-cd40-4782-b31e-086b1aa66312-kube-api-access-7rc6z\") pod \"cluster-monitoring-operator-6f55c4f58d-c8dsw\" (UID: \"81b34b10-cd40-4782-b31e-086b1aa66312\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:46:47.644979 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.644935    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-mfkbb\" (UID: \"e87d9312-d1ae-4934-b93c-6c53272c30a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"
Jul 31 23:46:47.644979 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.644959    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59605a10-07f0-4484-bdda-7779bc9c8316-registry-certificates\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.645076 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.644984    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59605a10-07f0-4484-bdda-7779bc9c8316-installation-pull-secrets\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.645076 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.645024    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.645076 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.645054    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59605a10-07f0-4484-bdda-7779bc9c8316-ca-trust-extracted\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.645217 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.645099    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/81b34b10-cd40-4782-b31e-086b1aa66312-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-c8dsw\" (UID: \"81b34b10-cd40-4782-b31e-086b1aa66312\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:46:47.645217 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.645130    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mjkp\" (UniqueName: \"kubernetes.io/projected/e87d9312-d1ae-4934-b93c-6c53272c30a4-kube-api-access-8mjkp\") pod \"cluster-samples-operator-559cbf5745-mfkbb\" (UID: \"e87d9312-d1ae-4934-b93c-6c53272c30a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"
Jul 31 23:46:47.645217 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.645155    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59605a10-07f0-4484-bdda-7779bc9c8316-image-registry-private-configuration\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.655649 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.655629    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s"
Jul 31 23:46:47.663068 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.663042    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:47.745749 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.745717    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-bound-sa-token\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.745888 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.745763    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkgw4\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-kube-api-access-xkgw4\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.745952 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.745884    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59605a10-07f0-4484-bdda-7779bc9c8316-trusted-ca\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.745952 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.745937    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-c8dsw\" (UID: \"81b34b10-cd40-4782-b31e-086b1aa66312\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:46:47.746054 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.745989    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rc6z\" (UniqueName: \"kubernetes.io/projected/81b34b10-cd40-4782-b31e-086b1aa66312-kube-api-access-7rc6z\") pod \"cluster-monitoring-operator-6f55c4f58d-c8dsw\" (UID: \"81b34b10-cd40-4782-b31e-086b1aa66312\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:46:47.746054 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:47.746034    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:47.746150 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.746054    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-mfkbb\" (UID: \"e87d9312-d1ae-4934-b93c-6c53272c30a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"
Jul 31 23:46:47.746150 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:47.746113    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls podName:81b34b10-cd40-4782-b31e-086b1aa66312 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:48.246092617 +0000 UTC m=+38.705555418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-c8dsw" (UID: "81b34b10-cd40-4782-b31e-086b1aa66312") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:47.746275 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:47.746151    2546 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:46:47.746275 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.746186    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59605a10-07f0-4484-bdda-7779bc9c8316-registry-certificates\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.746275 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:47.746199    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls podName:e87d9312-d1ae-4934-b93c-6c53272c30a4 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:48.246183626 +0000 UTC m=+38.705646428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-mfkbb" (UID: "e87d9312-d1ae-4934-b93c-6c53272c30a4") : secret "samples-operator-tls" not found
Jul 31 23:46:47.746275 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.746224    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59605a10-07f0-4484-bdda-7779bc9c8316-installation-pull-secrets\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.746473 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.746276    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.746473 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.746323    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59605a10-07f0-4484-bdda-7779bc9c8316-ca-trust-extracted\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.746473 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:47.746384    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:47.746473 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:47.746398    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b94dfb9-cngmx: secret "image-registry-tls" not found
Jul 31 23:46:47.746473 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.746407    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/81b34b10-cd40-4782-b31e-086b1aa66312-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-c8dsw\" (UID: \"81b34b10-cd40-4782-b31e-086b1aa66312\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:46:47.746473 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:47.746439    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls podName:59605a10-07f0-4484-bdda-7779bc9c8316 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:48.246425811 +0000 UTC m=+38.705888621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls") pod "image-registry-5f6b94dfb9-cngmx" (UID: "59605a10-07f0-4484-bdda-7779bc9c8316") : secret "image-registry-tls" not found
Jul 31 23:46:47.746473 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.746462    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjkp\" (UniqueName: \"kubernetes.io/projected/e87d9312-d1ae-4934-b93c-6c53272c30a4-kube-api-access-8mjkp\") pod \"cluster-samples-operator-559cbf5745-mfkbb\" (UID: \"e87d9312-d1ae-4934-b93c-6c53272c30a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"
Jul 31 23:46:47.746786 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.746494    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59605a10-07f0-4484-bdda-7779bc9c8316-image-registry-private-configuration\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.746786 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.746745    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59605a10-07f0-4484-bdda-7779bc9c8316-registry-certificates\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.746991 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.746930    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59605a10-07f0-4484-bdda-7779bc9c8316-ca-trust-extracted\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.747098 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.747031    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59605a10-07f0-4484-bdda-7779bc9c8316-trusted-ca\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.747098 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.747061    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/81b34b10-cd40-4782-b31e-086b1aa66312-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-c8dsw\" (UID: \"81b34b10-cd40-4782-b31e-086b1aa66312\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:46:47.749305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.749242    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59605a10-07f0-4484-bdda-7779bc9c8316-installation-pull-secrets\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.749305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.749279    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59605a10-07f0-4484-bdda-7779bc9c8316-image-registry-private-configuration\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.756565 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.756529    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rc6z\" (UniqueName: \"kubernetes.io/projected/81b34b10-cd40-4782-b31e-086b1aa66312-kube-api-access-7rc6z\") pod \"cluster-monitoring-operator-6f55c4f58d-c8dsw\" (UID: \"81b34b10-cd40-4782-b31e-086b1aa66312\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:46:47.756853 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.756800    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-bound-sa-token\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.757107 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.757031    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkgw4\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-kube-api-access-xkgw4\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:47.757749 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.757722    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjkp\" (UniqueName: \"kubernetes.io/projected/e87d9312-d1ae-4934-b93c-6c53272c30a4-kube-api-access-8mjkp\") pod \"cluster-samples-operator-559cbf5745-mfkbb\" (UID: \"e87d9312-d1ae-4934-b93c-6c53272c30a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"
Jul 31 23:46:47.766402 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.766384    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-ktsf2"
Jul 31 23:46:47.787781 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.787757    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s"]
Jul 31 23:46:47.790688 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:47.790662    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e582be6_7c99_42ca_ad13_a97315764e09.slice/crio-21db8684f176fe61d4c42434fd9688ec017b2b5e90b59865fb146d2a559dfa05 WatchSource:0}: Error finding container 21db8684f176fe61d4c42434fd9688ec017b2b5e90b59865fb146d2a559dfa05: Status 404 returned error can't find the container with id 21db8684f176fe61d4c42434fd9688ec017b2b5e90b59865fb146d2a559dfa05
Jul 31 23:46:47.907601 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:47.907362    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-ktsf2"]
Jul 31 23:46:48.049245 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:48.049211    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:48.049407 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:48.049285    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:48.049407 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:48.049368    2546 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:48.049512 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:48.049410    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle podName:e005be7e-fed7-468e-9a7d-601db46f362f nodeName:}" failed. No retries permitted until 2026-07-31 23:46:49.049394189 +0000 UTC m=+39.508856984 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle") pod "router-default-f44c6f5f8-5hkk9" (UID: "e005be7e-fed7-468e-9a7d-601db46f362f") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:48.049512 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:48.049429    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs podName:e005be7e-fed7-468e-9a7d-601db46f362f nodeName:}" failed. No retries permitted until 2026-07-31 23:46:49.049420729 +0000 UTC m=+39.508883524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs") pod "router-default-f44c6f5f8-5hkk9" (UID: "e005be7e-fed7-468e-9a7d-601db46f362f") : secret "router-metrics-certs-default" not found
Jul 31 23:46:48.250684 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:48.250606    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-mfkbb\" (UID: \"e87d9312-d1ae-4934-b93c-6c53272c30a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"
Jul 31 23:46:48.250684 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:48.250650    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:48.250902 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:48.250791    2546 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:46:48.250902 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:48.250802    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:48.250902 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:48.250867    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls podName:e87d9312-d1ae-4934-b93c-6c53272c30a4 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:49.250846161 +0000 UTC m=+39.710308954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-mfkbb" (UID: "e87d9312-d1ae-4934-b93c-6c53272c30a4") : secret "samples-operator-tls" not found
Jul 31 23:46:48.250902 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:48.250877    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b94dfb9-cngmx: secret "image-registry-tls" not found
Jul 31 23:46:48.251093 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:48.250900    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-c8dsw\" (UID: \"81b34b10-cd40-4782-b31e-086b1aa66312\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:46:48.251093 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:48.250929    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls podName:59605a10-07f0-4484-bdda-7779bc9c8316 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:49.250912391 +0000 UTC m=+39.710375182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls") pod "image-registry-5f6b94dfb9-cngmx" (UID: "59605a10-07f0-4484-bdda-7779bc9c8316") : secret "image-registry-tls" not found
Jul 31 23:46:48.251093 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:48.250977    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:48.251093 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:48.251027    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls podName:81b34b10-cd40-4782-b31e-086b1aa66312 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:49.251015559 +0000 UTC m=+39.710478367 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-c8dsw" (UID: "81b34b10-cd40-4782-b31e-086b1aa66312") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:48.351098 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:48.351060    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s" event={"ID":"4e582be6-7c99-42ca-ad13-a97315764e09","Type":"ContainerStarted","Data":"21db8684f176fe61d4c42434fd9688ec017b2b5e90b59865fb146d2a559dfa05"}
Jul 31 23:46:48.354995 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:48.354967    2546 generic.go:356] "Generic (PLEG): container finished" podID="a66649bd-a671-4bee-af65-7930bfc49e55" containerID="86ca1944d5f8e9e1a16f2fde347faf74b1050a82e380861c21c2bfda79ab6226" exitCode=0
Jul 31 23:46:48.355133 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:48.355024    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4q4d7" event={"ID":"a66649bd-a671-4bee-af65-7930bfc49e55","Type":"ContainerDied","Data":"86ca1944d5f8e9e1a16f2fde347faf74b1050a82e380861c21c2bfda79ab6226"}
Jul 31 23:46:48.607186 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:46:48.606990    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c85d11a_d782_4c52_9126_d9b563ed453f.slice/crio-897b1c67eb80f4429bc775549ddba4c88cb1517cd3e4f35d423579d064ff7b42 WatchSource:0}: Error finding container 897b1c67eb80f4429bc775549ddba4c88cb1517cd3e4f35d423579d064ff7b42: Status 404 returned error can't find the container with id 897b1c67eb80f4429bc775549ddba4c88cb1517cd3e4f35d423579d064ff7b42
Jul 31 23:46:48.654759 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:48.654730    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:48.654860 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:48.654838    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:48.654908 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:48.654891    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls podName:fec5df48-6c8a-49ff-bb01-43d1d87ca8fd nodeName:}" failed. No retries permitted until 2026-07-31 23:46:52.654877018 +0000 UTC m=+43.114339810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls") pod "dns-default-jw8jw" (UID: "fec5df48-6c8a-49ff-bb01-43d1d87ca8fd") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:49.057919 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:49.057876    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:49.058091 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:49.058038    2546 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:49.058091 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:49.058041    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:49.058191 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:49.058106    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs podName:e005be7e-fed7-468e-9a7d-601db46f362f nodeName:}" failed. No retries permitted until 2026-07-31 23:46:51.05808729 +0000 UTC m=+41.517550100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs") pod "router-default-f44c6f5f8-5hkk9" (UID: "e005be7e-fed7-468e-9a7d-601db46f362f") : secret "router-metrics-certs-default" not found
Jul 31 23:46:49.058191 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:49.058144    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle podName:e005be7e-fed7-468e-9a7d-601db46f362f nodeName:}" failed. No retries permitted until 2026-07-31 23:46:51.058127687 +0000 UTC m=+41.517590483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle") pod "router-default-f44c6f5f8-5hkk9" (UID: "e005be7e-fed7-468e-9a7d-601db46f362f") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:49.260525 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:49.260449    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:49.260682 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:49.260549    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-c8dsw\" (UID: \"81b34b10-cd40-4782-b31e-086b1aa66312\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:46:49.260682 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:49.260592    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-mfkbb\" (UID: \"e87d9312-d1ae-4934-b93c-6c53272c30a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"
Jul 31 23:46:49.260682 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:49.260636    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:49.260682 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:49.260656    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b94dfb9-cngmx: secret "image-registry-tls" not found
Jul 31 23:46:49.260682 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:49.260673    2546 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:46:49.260929 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:49.260717    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls podName:59605a10-07f0-4484-bdda-7779bc9c8316 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:51.260697239 +0000 UTC m=+41.720160061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls") pod "image-registry-5f6b94dfb9-cngmx" (UID: "59605a10-07f0-4484-bdda-7779bc9c8316") : secret "image-registry-tls" not found
Jul 31 23:46:49.260929 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:49.260737    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls podName:e87d9312-d1ae-4934-b93c-6c53272c30a4 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:51.260726867 +0000 UTC m=+41.720189666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-mfkbb" (UID: "e87d9312-d1ae-4934-b93c-6c53272c30a4") : secret "samples-operator-tls" not found
Jul 31 23:46:49.260929 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:49.260756    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:49.260929 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:49.260783    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls podName:81b34b10-cd40-4782-b31e-086b1aa66312 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:51.260772058 +0000 UTC m=+41.720234852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-c8dsw" (UID: "81b34b10-cd40-4782-b31e-086b1aa66312") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:49.361877 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:49.361765    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7m7vf" event={"ID":"447bf0cf-4705-4db9-a4ba-cc6d257d3170","Type":"ContainerStarted","Data":"b957f5d64a8598b6b2e1cf519365345bafef318acc9fed4bf3baae0c2748cab0"}
Jul 31 23:46:49.363804 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:49.363750    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-ktsf2" event={"ID":"8c85d11a-d782-4c52-9126-d9b563ed453f","Type":"ContainerStarted","Data":"897b1c67eb80f4429bc775549ddba4c88cb1517cd3e4f35d423579d064ff7b42"}
Jul 31 23:46:49.368980 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:49.368463    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4q4d7" event={"ID":"a66649bd-a671-4bee-af65-7930bfc49e55","Type":"ContainerStarted","Data":"e03d2125e3c8194682ba075479da6dfb9934ac2411ddb4b312d85f5b6fe84074"}
Jul 31 23:46:49.379557 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:49.379510    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7m7vf" podStartSLOduration=3.166541117 podStartE2EDuration="5.379493223s" totalImagesPullingTime="2.212952106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:49.378114925 +0000 UTC m=+39.837577737" watchObservedRunningTime="2026-07-31 23:46:49.379493223 +0000 UTC m=+39.838956038"
Jul 31 23:46:49.401057 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:49.400087    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4q4d7" podStartSLOduration=10.085064443 podStartE2EDuration="39.400067327s" totalImagesPullingTime="29.315002884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:49.398433282 +0000 UTC m=+39.857896095" watchObservedRunningTime="2026-07-31 23:46:49.400067327 +0000 UTC m=+39.859530141"
Jul 31 23:46:49.520109 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:49.520015    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:49.576955 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:49.576924    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:49.931668 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:49.931566    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:50.032753 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:50.032717    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:51.077570 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:51.077532    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:51.078020 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:51.077718    2546 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:51.078020 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:51.077718    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:51.078020 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:51.077802    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs podName:e005be7e-fed7-468e-9a7d-601db46f362f nodeName:}" failed. No retries permitted until 2026-07-31 23:46:55.077780635 +0000 UTC m=+45.537243443 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs") pod "router-default-f44c6f5f8-5hkk9" (UID: "e005be7e-fed7-468e-9a7d-601db46f362f") : secret "router-metrics-certs-default" not found
Jul 31 23:46:51.078020 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:51.077822    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle podName:e005be7e-fed7-468e-9a7d-601db46f362f nodeName:}" failed. No retries permitted until 2026-07-31 23:46:55.077815818 +0000 UTC m=+45.537278609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle") pod "router-default-f44c6f5f8-5hkk9" (UID: "e005be7e-fed7-468e-9a7d-601db46f362f") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:51.279743 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:51.279700    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-mfkbb\" (UID: \"e87d9312-d1ae-4934-b93c-6c53272c30a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"
Jul 31 23:46:51.279918 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:51.279761    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:51.279918 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:51.279878    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-c8dsw\" (UID: \"81b34b10-cd40-4782-b31e-086b1aa66312\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:46:51.279918 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:51.279890    2546 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:46:51.280080 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:51.279934    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:51.280080 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:51.279953    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b94dfb9-cngmx: secret "image-registry-tls" not found
Jul 31 23:46:51.280080 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:51.279972    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls podName:e87d9312-d1ae-4934-b93c-6c53272c30a4 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:55.279948655 +0000 UTC m=+45.739411466 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-mfkbb" (UID: "e87d9312-d1ae-4934-b93c-6c53272c30a4") : secret "samples-operator-tls" not found
Jul 31 23:46:51.280080 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:51.280001    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls podName:59605a10-07f0-4484-bdda-7779bc9c8316 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:55.279985848 +0000 UTC m=+45.739448643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls") pod "image-registry-5f6b94dfb9-cngmx" (UID: "59605a10-07f0-4484-bdda-7779bc9c8316") : secret "image-registry-tls" not found
Jul 31 23:46:51.280080 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:51.280027    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:51.280300 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:51.280087    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls podName:81b34b10-cd40-4782-b31e-086b1aa66312 nodeName:}" failed. No retries permitted until 2026-07-31 23:46:55.280069784 +0000 UTC m=+45.739532577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-c8dsw" (UID: "81b34b10-cd40-4782-b31e-086b1aa66312") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:51.374567 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:51.374529    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s" event={"ID":"4e582be6-7c99-42ca-ad13-a97315764e09","Type":"ContainerStarted","Data":"5dc2c0c29a1462e8c5f72a0c85c6e79a497c1daa646d8a2e97bd1e4e7254d71b"}
Jul 31 23:46:51.394044 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:51.393998    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s" podStartSLOduration=1.61469748 podStartE2EDuration="4.393965529s" totalImagesPullingTime="2.779268049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:51.392516192 +0000 UTC m=+41.851979033" watchObservedRunningTime="2026-07-31 23:46:51.393965529 +0000 UTC m=+41.853428407"
Jul 31 23:46:52.226247 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:52.226217    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:52.300500 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:52.300468    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:52.378210 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:52.378176    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-ktsf2" event={"ID":"8c85d11a-d782-4c52-9126-d9b563ed453f","Type":"ContainerStarted","Data":"6a34e1e2f2c39153886abd3580a2920335c4821f5e91e657543f626e288c209a"}
Jul 31 23:46:52.691634 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:52.691564    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:46:52.691769 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:52.691675    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:46:52.691769 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:52.691730    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls podName:fec5df48-6c8a-49ff-bb01-43d1d87ca8fd nodeName:}" failed. No retries permitted until 2026-07-31 23:47:00.69171339 +0000 UTC m=+51.151176182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls") pod "dns-default-jw8jw" (UID: "fec5df48-6c8a-49ff-bb01-43d1d87ca8fd") : secret "dns-default-metrics-tls" not found
Jul 31 23:46:52.747832 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:52.747800    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:52.883274 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:52.883229    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:53.051646 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.051542    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:53.315561 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.315494    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:53.330666 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.330611    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-ktsf2" podStartSLOduration=2.862086421 podStartE2EDuration="6.330594711s" totalImagesPullingTime="3.46850829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:46:52.409941407 +0000 UTC m=+42.869404224" watchObservedRunningTime="2026-07-31 23:46:53.330594711 +0000 UTC m=+43.790057529"
Jul 31 23:46:53.331018 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.331002    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jkssq"]
Jul 31 23:46:53.347441 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.347419    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:53.359338 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.359315    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 23:46:53.359441 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.359316    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 23:46:53.359513 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.359470    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4rmhk\""
Jul 31 23:46:53.360407 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.360392    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 23:46:53.364143 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.364128    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 23:46:53.368705 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.368686    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jkssq"]
Jul 31 23:46:53.399279 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.398099    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eefe175d-a2df-4722-99af-40719128fc4d-data-volume\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:53.399279 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.398193    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eefe175d-a2df-4722-99af-40719128fc4d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:53.399279 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.398266    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:53.399279 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.398311    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmzch\" (UniqueName: \"kubernetes.io/projected/eefe175d-a2df-4722-99af-40719128fc4d-kube-api-access-jmzch\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:53.399279 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.398382    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eefe175d-a2df-4722-99af-40719128fc4d-crio-socket\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:53.445748 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.445703    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:53.499526 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.499495    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eefe175d-a2df-4722-99af-40719128fc4d-data-volume\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:53.499665 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.499643    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eefe175d-a2df-4722-99af-40719128fc4d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:53.499724 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.499701    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:53.499777 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.499746    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmzch\" (UniqueName: \"kubernetes.io/projected/eefe175d-a2df-4722-99af-40719128fc4d-kube-api-access-jmzch\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:53.499835 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:53.499769    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 23:46:53.499835 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.499806    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eefe175d-a2df-4722-99af-40719128fc4d-crio-socket\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:53.499835 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:53.499825    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:53.499952 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.499852    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eefe175d-a2df-4722-99af-40719128fc4d-data-volume\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:53.499952 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:53.499833    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eefe175d-a2df-4722-99af-40719128fc4d-kube-rbac-proxy-cm podName:eefe175d-a2df-4722-99af-40719128fc4d nodeName:}" failed. No retries permitted until 2026-07-31 23:46:53.999812288 +0000 UTC m=+44.459275086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eefe175d-a2df-4722-99af-40719128fc4d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jkssq" (UID: "eefe175d-a2df-4722-99af-40719128fc4d") : configmap "kube-rbac-proxy" not found
Jul 31 23:46:53.499952 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:53.499917    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls podName:eefe175d-a2df-4722-99af-40719128fc4d nodeName:}" failed. No retries permitted until 2026-07-31 23:46:53.999902536 +0000 UTC m=+44.459365336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jkssq" (UID: "eefe175d-a2df-4722-99af-40719128fc4d") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:53.500076 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.499972    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eefe175d-a2df-4722-99af-40719128fc4d-crio-socket\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:53.514032 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:53.514008    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmzch\" (UniqueName: \"kubernetes.io/projected/eefe175d-a2df-4722-99af-40719128fc4d-kube-api-access-jmzch\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:54.004666 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:54.004634    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eefe175d-a2df-4722-99af-40719128fc4d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:54.004859 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:54.004694    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:54.004859 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:54.004760    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 23:46:54.004859 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:54.004774    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:54.004859 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:54.004838    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eefe175d-a2df-4722-99af-40719128fc4d-kube-rbac-proxy-cm podName:eefe175d-a2df-4722-99af-40719128fc4d nodeName:}" failed. No retries permitted until 2026-07-31 23:46:55.004822333 +0000 UTC m=+45.464285128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eefe175d-a2df-4722-99af-40719128fc4d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jkssq" (UID: "eefe175d-a2df-4722-99af-40719128fc4d") : configmap "kube-rbac-proxy" not found
Jul 31 23:46:54.005022 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:54.004900    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls podName:eefe175d-a2df-4722-99af-40719128fc4d nodeName:}" failed. No retries permitted until 2026-07-31 23:46:55.004884475 +0000 UTC m=+45.464347267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jkssq" (UID: "eefe175d-a2df-4722-99af-40719128fc4d") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:54.597825 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:54.597791    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:54.674469 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:54.674436    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:55.013950 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:55.013898    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eefe175d-a2df-4722-99af-40719128fc4d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:55.014154 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:55.014006    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:55.014154 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:55.014024    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 23:46:55.014154 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:55.014079    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eefe175d-a2df-4722-99af-40719128fc4d-kube-rbac-proxy-cm podName:eefe175d-a2df-4722-99af-40719128fc4d nodeName:}" failed. No retries permitted until 2026-07-31 23:46:57.014065539 +0000 UTC m=+47.473528329 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eefe175d-a2df-4722-99af-40719128fc4d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jkssq" (UID: "eefe175d-a2df-4722-99af-40719128fc4d") : configmap "kube-rbac-proxy" not found
Jul 31 23:46:55.014154 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:55.014152    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:55.014436 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:55.014204    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls podName:eefe175d-a2df-4722-99af-40719128fc4d nodeName:}" failed. No retries permitted until 2026-07-31 23:46:57.014186278 +0000 UTC m=+47.473649072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jkssq" (UID: "eefe175d-a2df-4722-99af-40719128fc4d") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:55.014564 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:55.014534    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:55.087074 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:55.087044    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:55.115184 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:55.115163    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:55.115266 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:55.115205    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:46:55.115348 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:55.115314    2546 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 31 23:46:55.115348 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:55.115323    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle podName:e005be7e-fed7-468e-9a7d-601db46f362f nodeName:}" failed. No retries permitted until 2026-07-31 23:47:03.115308503 +0000 UTC m=+53.574771307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle") pod "router-default-f44c6f5f8-5hkk9" (UID: "e005be7e-fed7-468e-9a7d-601db46f362f") : configmap references non-existent config key: service-ca.crt
Jul 31 23:46:55.115419 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:55.115353    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs podName:e005be7e-fed7-468e-9a7d-601db46f362f nodeName:}" failed. No retries permitted until 2026-07-31 23:47:03.115341931 +0000 UTC m=+53.574804723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs") pod "router-default-f44c6f5f8-5hkk9" (UID: "e005be7e-fed7-468e-9a7d-601db46f362f") : secret "router-metrics-certs-default" not found
Jul 31 23:46:55.315925 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:55.315861    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:46:55.316070 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:55.315941    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-c8dsw\" (UID: \"81b34b10-cd40-4782-b31e-086b1aa66312\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:46:55.316070 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:55.315994    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 23:46:55.316070 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:55.316010    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f6b94dfb9-cngmx: secret "image-registry-tls" not found
Jul 31 23:46:55.316070 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:55.316036    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:55.316070 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:55.316051    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls podName:59605a10-07f0-4484-bdda-7779bc9c8316 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:03.31603858 +0000 UTC m=+53.775501375 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls") pod "image-registry-5f6b94dfb9-cngmx" (UID: "59605a10-07f0-4484-bdda-7779bc9c8316") : secret "image-registry-tls" not found
Jul 31 23:46:55.316070 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:55.315999    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-mfkbb\" (UID: \"e87d9312-d1ae-4934-b93c-6c53272c30a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"
Jul 31 23:46:55.316070 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:55.316050    2546 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 23:46:55.316355 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:55.316107    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls podName:81b34b10-cd40-4782-b31e-086b1aa66312 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:03.316091513 +0000 UTC m=+53.775554327 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-c8dsw" (UID: "81b34b10-cd40-4782-b31e-086b1aa66312") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:46:55.316355 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:55.316128    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls podName:e87d9312-d1ae-4934-b93c-6c53272c30a4 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:03.316117953 +0000 UTC m=+53.775580773 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-mfkbb" (UID: "e87d9312-d1ae-4934-b93c-6c53272c30a4") : secret "samples-operator-tls" not found
Jul 31 23:46:55.686682 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:55.686646    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:56.027383 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:56.027317    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v58mz_655fd51c-e55e-4fa9-8a11-8de5f622b670/dns-node-resolver/0.log"
Jul 31 23:46:56.322276 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:56.322172    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:56.822444 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:56.822416    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hp4zj_5278364c-d649-48ba-8388-d277b82f21a7/node-ca/0.log"
Jul 31 23:46:57.029465 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:57.029430    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eefe175d-a2df-4722-99af-40719128fc4d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:57.029629 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:57.029479    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:57.029674 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:57.029631    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:57.029719 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:46:57.029691    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls podName:eefe175d-a2df-4722-99af-40719128fc4d nodeName:}" failed. No retries permitted until 2026-07-31 23:47:01.029676528 +0000 UTC m=+51.489139319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jkssq" (UID: "eefe175d-a2df-4722-99af-40719128fc4d") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:46:57.030029 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:57.030013    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eefe175d-a2df-4722-99af-40719128fc4d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:46:57.622046 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:57.622012    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7m7vf_447bf0cf-4705-4db9-a4ba-cc6d257d3170/serve-healthcheck-canary/0.log"
Jul 31 23:46:58.357743 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:58.357705    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:58.506464 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:58.506423    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:58.669727 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:58.669690    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:58.770051 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:58.770025    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:58.872049 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:58.871984    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:58.959289 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:58.959204    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:59.035961 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:59.035931    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:46:59.822709 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:46:59.822675    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:00.214100 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:00.214069    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:00.420075 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:00.420039    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:00.760671 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:00.760636    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:47:00.760831 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:47:00.760785    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 23:47:00.760877 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:47:00.760853    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls podName:fec5df48-6c8a-49ff-bb01-43d1d87ca8fd nodeName:}" failed. No retries permitted until 2026-07-31 23:47:16.760836972 +0000 UTC m=+67.220299766 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls") pod "dns-default-jw8jw" (UID: "fec5df48-6c8a-49ff-bb01-43d1d87ca8fd") : secret "dns-default-metrics-tls" not found
Jul 31 23:47:00.811028 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:00.811002    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:01.063918 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:01.063848    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:47:01.064238 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:47:01.063987    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 23:47:01.064238 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:47:01.064043    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls podName:eefe175d-a2df-4722-99af-40719128fc4d nodeName:}" failed. No retries permitted until 2026-07-31 23:47:09.064028355 +0000 UTC m=+59.523491146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jkssq" (UID: "eefe175d-a2df-4722-99af-40719128fc4d") : secret "insights-runtime-extractor-tls" not found
Jul 31 23:47:01.415262 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:01.415233    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:02.017981 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:02.017941    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:02.079310 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:02.079278    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:02.212294 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:02.212264    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:02.626631 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:02.626596    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:02.824521 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:02.824493    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:02.895898 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:02.895829    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:03.179759 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:03.179687    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:47:03.179759 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:03.179729    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:47:03.180220 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:47:03.179852    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle podName:e005be7e-fed7-468e-9a7d-601db46f362f nodeName:}" failed. No retries permitted until 2026-07-31 23:47:19.179838497 +0000 UTC m=+69.639301288 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle") pod "router-default-f44c6f5f8-5hkk9" (UID: "e005be7e-fed7-468e-9a7d-601db46f362f") : configmap references non-existent config key: service-ca.crt
Jul 31 23:47:03.182026 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:03.182001    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e005be7e-fed7-468e-9a7d-601db46f362f-metrics-certs\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:47:03.217705 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:03.217678    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:03.382065 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:03.382033    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-c8dsw\" (UID: \"81b34b10-cd40-4782-b31e-086b1aa66312\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:47:03.382216 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:03.382094    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-mfkbb\" (UID: \"e87d9312-d1ae-4934-b93c-6c53272c30a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"
Jul 31 23:47:03.382216 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:03.382115    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:47:03.382216 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:47:03.382199    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 23:47:03.382346 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:47:03.382295    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls podName:81b34b10-cd40-4782-b31e-086b1aa66312 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:19.382276565 +0000 UTC m=+69.841739357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-c8dsw" (UID: "81b34b10-cd40-4782-b31e-086b1aa66312") : secret "cluster-monitoring-operator-tls" not found
Jul 31 23:47:03.384435 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:03.384409    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls\") pod \"image-registry-5f6b94dfb9-cngmx\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") " pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:47:03.384555 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:03.384478    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e87d9312-d1ae-4934-b93c-6c53272c30a4-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-mfkbb\" (UID: \"e87d9312-d1ae-4934-b93c-6c53272c30a4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"
Jul 31 23:47:03.469293 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:03.469210    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"
Jul 31 23:47:03.487491 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:03.487468    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:47:03.618528 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:03.618496    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb"]
Jul 31 23:47:03.645578 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:03.645552    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f6b94dfb9-cngmx"]
Jul 31 23:47:03.648597 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:03.648571    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59605a10_07f0_4484_bdda_7779bc9c8316.slice/crio-57f5bfe6d6f9599f17c0c2dd4b6d79c7daaeb43b0003e508e73b7e4af2c9a959 WatchSource:0}: Error finding container 57f5bfe6d6f9599f17c0c2dd4b6d79c7daaeb43b0003e508e73b7e4af2c9a959: Status 404 returned error can't find the container with id 57f5bfe6d6f9599f17c0c2dd4b6d79c7daaeb43b0003e508e73b7e4af2c9a959
Jul 31 23:47:04.017568 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:04.017496    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:04.222605 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:04.222572    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:04.408558 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:04.408461    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx" event={"ID":"59605a10-07f0-4484-bdda-7779bc9c8316","Type":"ContainerStarted","Data":"af9fe373b0a67c2535b9600e0977260c72e6a7d90ea1d5e90ebd58cee757787b"}
Jul 31 23:47:04.408558 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:04.408522    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx" event={"ID":"59605a10-07f0-4484-bdda-7779bc9c8316","Type":"ContainerStarted","Data":"57f5bfe6d6f9599f17c0c2dd4b6d79c7daaeb43b0003e508e73b7e4af2c9a959"}
Jul 31 23:47:04.408799 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:04.408626    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:47:04.409682 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:04.409653    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb" event={"ID":"e87d9312-d1ae-4934-b93c-6c53272c30a4","Type":"ContainerStarted","Data":"e1bf61a6c1e4140db80c870456507902706e9321351474ca91ebffb06a61009b"}
Jul 31 23:47:04.427824 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:04.427775    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx" podStartSLOduration=17.427758693 podStartE2EDuration="17.427758693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:04.426806241 +0000 UTC m=+54.886269057" watchObservedRunningTime="2026-07-31 23:47:04.427758693 +0000 UTC m=+54.887221508"
Jul 31 23:47:04.599053 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:04.599012    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:04.670156 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:04.670061    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:04.819976 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:04.819941    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:05.415082 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:05.415036    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb" event={"ID":"e87d9312-d1ae-4934-b93c-6c53272c30a4","Type":"ContainerStarted","Data":"e7dbf480434e175edd9615300583f66e80fdfc59083635aad1d2e4407f9603b4"}
Jul 31 23:47:05.415494 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:05.415140    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb" event={"ID":"e87d9312-d1ae-4934-b93c-6c53272c30a4","Type":"ContainerStarted","Data":"c305f7874de97505671b18bc991d035c729c69a5d45c9cef8de1550856db2729"}
Jul 31 23:47:05.435022 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:05.434980    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-mfkbb" podStartSLOduration=16.895715501 podStartE2EDuration="18.434966889s" totalImagesPullingTime="1.539251388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:05.434142499 +0000 UTC m=+55.893605324" watchObservedRunningTime="2026-07-31 23:47:05.434966889 +0000 UTC m=+55.894429681"
Jul 31 23:47:05.440045 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:05.440024    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:05.615683 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:05.615644    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.024891 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:06.024809    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:06.832721 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:06.832692    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:07.029206 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:07.029177    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:07.415596 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:07.415563    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:07.615489 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:07.615461    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:08.006063 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:08.006011    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:08.064437 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:08.064384    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:08.213084 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:08.213059    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:08.629303 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:08.629270    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:08.820345 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:08.820307    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:09.027948 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:09.027870    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:09.126629 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:09.126594    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:47:09.128687 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:09.128668    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eefe175d-a2df-4722-99af-40719128fc4d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkssq\" (UID: \"eefe175d-a2df-4722-99af-40719128fc4d\") " pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:47:09.256209 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:09.256172    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jkssq"
Jul 31 23:47:09.374167 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:09.374138    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jkssq"]
Jul 31 23:47:09.377053 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:09.377028    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeefe175d_a2df_4722_99af_40719128fc4d.slice/crio-cef22d418b03a4caa30572fd8e6d4da3118b664f215f4ddca199d185a4157110 WatchSource:0}: Error finding container cef22d418b03a4caa30572fd8e6d4da3118b664f215f4ddca199d185a4157110: Status 404 returned error can't find the container with id cef22d418b03a4caa30572fd8e6d4da3118b664f215f4ddca199d185a4157110
Jul 31 23:47:09.427129 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:09.427014    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jkssq" event={"ID":"eefe175d-a2df-4722-99af-40719128fc4d","Type":"ContainerStarted","Data":"dcd8c1ee3cdf461c780e650057bf21b67c7aef6c6bc257d76f527793e3288dc3"}
Jul 31 23:47:09.427129 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:09.427062    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jkssq" event={"ID":"eefe175d-a2df-4722-99af-40719128fc4d","Type":"ContainerStarted","Data":"cef22d418b03a4caa30572fd8e6d4da3118b664f215f4ddca199d185a4157110"}
Jul 31 23:47:10.336356 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:10.336292    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2qmq9"
Jul 31 23:47:10.431265 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:10.431219    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jkssq" event={"ID":"eefe175d-a2df-4722-99af-40719128fc4d","Type":"ContainerStarted","Data":"0bebc530b8aa05e135793da3ddbabb674b7df747a3bab576d247e306009702e2"}
Jul 31 23:47:10.653245 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:10.653210    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:10.754184 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:10.754151    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:10.812244 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:10.812186    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:10.961038 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:10.960946    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:11.331394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:11.331311    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:11.432315 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:11.432279    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:11.738135 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:11.738104    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:11.801835 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:11.801798    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:11.938798 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:11.938773    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.027640 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:12.027612    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.438353 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:12.438319    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jkssq" event={"ID":"eefe175d-a2df-4722-99af-40719128fc4d","Type":"ContainerStarted","Data":"cc6789e89ae07b6126f77ff7842a2c49fce69223764e8a93d563329b46304b4a"}
Jul 31 23:47:12.462512 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:12.462469    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jkssq" podStartSLOduration=17.372297415 podStartE2EDuration="19.462454743s" totalImagesPullingTime="2.090157328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:12.461082042 +0000 UTC m=+62.920544856" watchObservedRunningTime="2026-07-31 23:47:12.462454743 +0000 UTC m=+62.921917552"
Jul 31 23:47:12.558435 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:12.558397    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.755309 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:12.755220    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:12.963114 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:12.963085    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:13.018888 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:13.018818    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:15.784305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.784248    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs\") pod \"network-metrics-daemon-5qcqv\" (UID: \"1fcbea3e-f254-440f-9281-bb84b6c5afaf\") " pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:47:15.784305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.784316    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:47:15.784735 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.784338    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbqms\" (UniqueName: \"kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms\") pod \"network-check-target-wr59z\" (UID: \"4c5d6736-af34-4032-9c3d-d7e95b856f36\") " pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:47:15.786980 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.786955    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbqms\" (UniqueName: \"kubernetes.io/projected/4c5d6736-af34-4032-9c3d-d7e95b856f36-kube-api-access-rbqms\") pod \"network-check-target-wr59z\" (UID: \"4c5d6736-af34-4032-9c3d-d7e95b856f36\") " pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:47:15.788196 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.788174    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 23:47:15.788317 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.788221    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 23:47:15.796629 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.796609    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4eb3a0b2-c573-43c7-a8dd-a01c3545d78b-original-pull-secret\") pod \"global-pull-secret-syncer-zmcw2\" (UID: \"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b\") " pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:47:15.796714 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.796698    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcbea3e-f254-440f-9281-bb84b6c5afaf-metrics-certs\") pod \"network-metrics-daemon-5qcqv\" (UID: \"1fcbea3e-f254-440f-9281-bb84b6c5afaf\") " pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:47:15.874953 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.874922    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f6b94dfb9-cngmx"]
Jul 31 23:47:15.902501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.902479    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65d74d959b-7g6nj"]
Jul 31 23:47:15.905167 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.905152    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:15.922023 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.922001    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65d74d959b-7g6nj"]
Jul 31 23:47:15.939813 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.939783    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:15.981909 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.981889    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmcw2"
Jul 31 23:47:15.986637 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.986620    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbab1df5-84fb-437a-8b11-6f04726b5b2e-ca-trust-extracted\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:15.986726 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.986645    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbab1df5-84fb-437a-8b11-6f04726b5b2e-trusted-ca\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:15.986726 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.986674    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbab1df5-84fb-437a-8b11-6f04726b5b2e-registry-certificates\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:15.986726 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.986708    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbab1df5-84fb-437a-8b11-6f04726b5b2e-bound-sa-token\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:15.986839 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.986744    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbab1df5-84fb-437a-8b11-6f04726b5b2e-registry-tls\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:15.986839 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.986763    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbab1df5-84fb-437a-8b11-6f04726b5b2e-image-registry-private-configuration\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:15.986839 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.986779    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twm6c\" (UniqueName: \"kubernetes.io/projected/fbab1df5-84fb-437a-8b11-6f04726b5b2e-kube-api-access-twm6c\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:15.986839 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.986830    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbab1df5-84fb-437a-8b11-6f04726b5b2e-installation-pull-secrets\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:15.989829 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.989809    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-wl8v7"]
Jul 31 23:47:15.991735 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.991675    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n5wdr\""
Jul 31 23:47:15.991735 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.991720    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-wl8v7"
Jul 31 23:47:15.994545 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.994527    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 31 23:47:15.994955 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.994940    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 31 23:47:15.996132 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.996116    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zhtdr\""
Jul 31 23:47:15.998537 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:15.998521    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:47:16.003643 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.003626    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fr2ph\""
Jul 31 23:47:16.004614 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.004601    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qcqv"
Jul 31 23:47:16.019268 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.019232    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-wl8v7"]
Jul 31 23:47:16.079697 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.079502    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:16.087754 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.087698    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbab1df5-84fb-437a-8b11-6f04726b5b2e-bound-sa-token\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.087892 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.087757    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbab1df5-84fb-437a-8b11-6f04726b5b2e-registry-tls\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.087892 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.087794    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbab1df5-84fb-437a-8b11-6f04726b5b2e-image-registry-private-configuration\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.087892 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.087821    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twm6c\" (UniqueName: \"kubernetes.io/projected/fbab1df5-84fb-437a-8b11-6f04726b5b2e-kube-api-access-twm6c\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.087892 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.087853    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbab1df5-84fb-437a-8b11-6f04726b5b2e-installation-pull-secrets\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.088116 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.087901    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq2rk\" (UniqueName: \"kubernetes.io/projected/7ed00006-4005-410f-b5f0-cc72951ee8b4-kube-api-access-xq2rk\") pod \"downloads-77c974ffcf-wl8v7\" (UID: \"7ed00006-4005-410f-b5f0-cc72951ee8b4\") " pod="openshift-console/downloads-77c974ffcf-wl8v7"
Jul 31 23:47:16.088116 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.087941    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbab1df5-84fb-437a-8b11-6f04726b5b2e-ca-trust-extracted\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.088116 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.087970    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbab1df5-84fb-437a-8b11-6f04726b5b2e-trusted-ca\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.088116 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.088007    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbab1df5-84fb-437a-8b11-6f04726b5b2e-registry-certificates\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.089115 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.088868    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbab1df5-84fb-437a-8b11-6f04726b5b2e-registry-certificates\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.089115 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.089033    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbab1df5-84fb-437a-8b11-6f04726b5b2e-ca-trust-extracted\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.089677 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.089651    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbab1df5-84fb-437a-8b11-6f04726b5b2e-trusted-ca\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.091309 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.091270    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbab1df5-84fb-437a-8b11-6f04726b5b2e-installation-pull-secrets\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.091428 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.091409    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbab1df5-84fb-437a-8b11-6f04726b5b2e-registry-tls\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.091922 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.091865    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbab1df5-84fb-437a-8b11-6f04726b5b2e-image-registry-private-configuration\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.097740 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.097727    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbab1df5-84fb-437a-8b11-6f04726b5b2e-bound-sa-token\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.098242 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.098216    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twm6c\" (UniqueName: \"kubernetes.io/projected/fbab1df5-84fb-437a-8b11-6f04726b5b2e-kube-api-access-twm6c\") pod \"image-registry-65d74d959b-7g6nj\" (UID: \"fbab1df5-84fb-437a-8b11-6f04726b5b2e\") " pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.133474 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.133427    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zmcw2"]
Jul 31 23:47:16.138264 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:16.138222    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb3a0b2_c573_43c7_a8dd_a01c3545d78b.slice/crio-b06640b803adc66edcf0a46ba2620131a1544beb79177dda35dce2a8b89feb66 WatchSource:0}: Error finding container b06640b803adc66edcf0a46ba2620131a1544beb79177dda35dce2a8b89feb66: Status 404 returned error can't find the container with id b06640b803adc66edcf0a46ba2620131a1544beb79177dda35dce2a8b89feb66
Jul 31 23:47:16.153845 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.153732    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wr59z"]
Jul 31 23:47:16.156284 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:16.156212    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5d6736_af34_4032_9c3d_d7e95b856f36.slice/crio-211cb9c26416e051753059447951efe416376638290971f4bafa83f675f57453 WatchSource:0}: Error finding container 211cb9c26416e051753059447951efe416376638290971f4bafa83f675f57453: Status 404 returned error can't find the container with id 211cb9c26416e051753059447951efe416376638290971f4bafa83f675f57453
Jul 31 23:47:16.173026 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.173005    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5qcqv"]
Jul 31 23:47:16.183247 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:16.183225    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcbea3e_f254_440f_9281_bb84b6c5afaf.slice/crio-75753404622dce12b773b5be32f908c945eb7b6bc5cbce5038099fa7cd285ced WatchSource:0}: Error finding container 75753404622dce12b773b5be32f908c945eb7b6bc5cbce5038099fa7cd285ced: Status 404 returned error can't find the container with id 75753404622dce12b773b5be32f908c945eb7b6bc5cbce5038099fa7cd285ced
Jul 31 23:47:16.188970 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.188945    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq2rk\" (UniqueName: \"kubernetes.io/projected/7ed00006-4005-410f-b5f0-cc72951ee8b4-kube-api-access-xq2rk\") pod \"downloads-77c974ffcf-wl8v7\" (UID: \"7ed00006-4005-410f-b5f0-cc72951ee8b4\") " pod="openshift-console/downloads-77c974ffcf-wl8v7"
Jul 31 23:47:16.200831 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.200812    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq2rk\" (UniqueName: \"kubernetes.io/projected/7ed00006-4005-410f-b5f0-cc72951ee8b4-kube-api-access-xq2rk\") pod \"downloads-77c974ffcf-wl8v7\" (UID: \"7ed00006-4005-410f-b5f0-cc72951ee8b4\") " pod="openshift-console/downloads-77c974ffcf-wl8v7"
Jul 31 23:47:16.217111 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.217091    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.309892 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.309864    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-wl8v7"
Jul 31 23:47:16.347485 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.347455    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65d74d959b-7g6nj"]
Jul 31 23:47:16.349994 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:16.349968    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbab1df5_84fb_437a_8b11_6f04726b5b2e.slice/crio-c109f67ef0a4299f93ac1a489551dc67e8ab3c3d6bf568823cb1f88c74251c42 WatchSource:0}: Error finding container c109f67ef0a4299f93ac1a489551dc67e8ab3c3d6bf568823cb1f88c74251c42: Status 404 returned error can't find the container with id c109f67ef0a4299f93ac1a489551dc67e8ab3c3d6bf568823cb1f88c74251c42
Jul 31 23:47:16.440270 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.440226    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-wl8v7"]
Jul 31 23:47:16.444396 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:16.444372    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed00006_4005_410f_b5f0_cc72951ee8b4.slice/crio-b06c65d1a25e11336f7556fe1863d0603bbb0096cc1327487c3438495e7105ed WatchSource:0}: Error finding container b06c65d1a25e11336f7556fe1863d0603bbb0096cc1327487c3438495e7105ed: Status 404 returned error can't find the container with id b06c65d1a25e11336f7556fe1863d0603bbb0096cc1327487c3438495e7105ed
Jul 31 23:47:16.451920 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.451895    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d74d959b-7g6nj" event={"ID":"fbab1df5-84fb-437a-8b11-6f04726b5b2e","Type":"ContainerStarted","Data":"76eef29b1f37ddda678b58a5bd7e77b6a887f024a41d60afe63e613d35f300b4"}
Jul 31 23:47:16.452030 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.451937    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d74d959b-7g6nj" event={"ID":"fbab1df5-84fb-437a-8b11-6f04726b5b2e","Type":"ContainerStarted","Data":"c109f67ef0a4299f93ac1a489551dc67e8ab3c3d6bf568823cb1f88c74251c42"}
Jul 31 23:47:16.452030 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.452004    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:16.453561 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.453530    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wr59z" event={"ID":"4c5d6736-af34-4032-9c3d-d7e95b856f36","Type":"ContainerStarted","Data":"16ddc6e54bc29a6b5c6ee0f23f6ca14f88c99edc626605c481348bae4b0f260b"}
Jul 31 23:47:16.453702 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.453581    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wr59z" event={"ID":"4c5d6736-af34-4032-9c3d-d7e95b856f36","Type":"ContainerStarted","Data":"211cb9c26416e051753059447951efe416376638290971f4bafa83f675f57453"}
Jul 31 23:47:16.453702 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.453607    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:47:16.454681 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.454662    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zmcw2" event={"ID":"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b","Type":"ContainerStarted","Data":"b06640b803adc66edcf0a46ba2620131a1544beb79177dda35dce2a8b89feb66"}
Jul 31 23:47:16.455644 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.455625    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qcqv" event={"ID":"1fcbea3e-f254-440f-9281-bb84b6c5afaf","Type":"ContainerStarted","Data":"75753404622dce12b773b5be32f908c945eb7b6bc5cbce5038099fa7cd285ced"}
Jul 31 23:47:16.479370 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.479321    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65d74d959b-7g6nj" podStartSLOduration=1.479303906 podStartE2EDuration="1.479303906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:16.477592669 +0000 UTC m=+66.937055486" watchObservedRunningTime="2026-07-31 23:47:16.479303906 +0000 UTC m=+66.938766718"
Jul 31 23:47:16.795718 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.795600    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:47:16.798534 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:16.798506    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fec5df48-6c8a-49ff-bb01-43d1d87ca8fd-metrics-tls\") pod \"dns-default-jw8jw\" (UID: \"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd\") " pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:47:17.045218 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:17.045182    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r2mdc\""
Jul 31 23:47:17.052747 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:17.052538    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:47:17.053101 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:17.053077    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:17.155216 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:17.155179    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:17.324142 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:17.324041    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wr59z" podStartSLOduration=67.324019481 podStartE2EDuration="1m7.324019481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:16.496029975 +0000 UTC m=+66.955492788" watchObservedRunningTime="2026-07-31 23:47:17.324019481 +0000 UTC m=+67.783482295"
Jul 31 23:47:17.325527 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:17.325459    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jw8jw"]
Jul 31 23:47:17.340951 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:17.340895    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec5df48_6c8a_49ff_bb01_43d1d87ca8fd.slice/crio-262be9df1a6a3fb828bd9e32f9e1a973d0ddc661a06555a631d47db19bde46d8 WatchSource:0}: Error finding container 262be9df1a6a3fb828bd9e32f9e1a973d0ddc661a06555a631d47db19bde46d8: Status 404 returned error can't find the container with id 262be9df1a6a3fb828bd9e32f9e1a973d0ddc661a06555a631d47db19bde46d8
Jul 31 23:47:17.395054 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:17.394887    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:17.459184 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:17.459130    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:17.466531 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:17.466423    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jw8jw" event={"ID":"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd","Type":"ContainerStarted","Data":"262be9df1a6a3fb828bd9e32f9e1a973d0ddc661a06555a631d47db19bde46d8"}
Jul 31 23:47:17.468469 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:17.468410    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-wl8v7" event={"ID":"7ed00006-4005-410f-b5f0-cc72951ee8b4","Type":"ContainerStarted","Data":"b06c65d1a25e11336f7556fe1863d0603bbb0096cc1327487c3438495e7105ed"}
Jul 31 23:47:18.475377 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:18.475238    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qcqv" event={"ID":"1fcbea3e-f254-440f-9281-bb84b6c5afaf","Type":"ContainerStarted","Data":"80c083e1a0adce747030a109801e7252967185e92f6c0d398f7e38e091c48686"}
Jul 31 23:47:18.475377 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:18.475321    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qcqv" event={"ID":"1fcbea3e-f254-440f-9281-bb84b6c5afaf","Type":"ContainerStarted","Data":"4288721e38ec79f4f9ca58fb45cffe6ce2d2ca73a31515bebc7bad2fa1684d73"}
Jul 31 23:47:18.493730 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:18.493563    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5qcqv" podStartSLOduration=67.322845371 podStartE2EDuration="1m8.493542927s" totalImagesPullingTime="1.170697556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:18.492446381 +0000 UTC m=+68.951909192" watchObservedRunningTime="2026-07-31 23:47:18.493542927 +0000 UTC m=+68.953005741"
Jul 31 23:47:18.576310 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:18.576276    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:18.638591 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:18.638505    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:18.979365 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:18.979285    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:19.041825 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:19.041796    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:19.106801 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:19.106744    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:19.219806 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:19.219768    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:47:19.220525 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:19.220502    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e005be7e-fed7-468e-9a7d-601db46f362f-service-ca-bundle\") pod \"router-default-f44c6f5f8-5hkk9\" (UID: \"e005be7e-fed7-468e-9a7d-601db46f362f\") " pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:47:19.259714 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:19.259651    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2cnf6\""
Jul 31 23:47:19.267977 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:19.267951    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:47:19.421413 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:19.421380    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-c8dsw\" (UID: \"81b34b10-cd40-4782-b31e-086b1aa66312\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:47:19.424496 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:19.424444    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/81b34b10-cd40-4782-b31e-086b1aa66312-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-c8dsw\" (UID: \"81b34b10-cd40-4782-b31e-086b1aa66312\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:47:19.651533 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:19.651502    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-f464j\""
Jul 31 23:47:19.659595 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:19.659569    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"
Jul 31 23:47:21.377177 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:21.377150    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw"]
Jul 31 23:47:21.382006 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:21.381842    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b34b10_cd40_4782_b31e_086b1aa66312.slice/crio-63fa0d2cafad11e02e495970c2280f22c5b4893287e2b1ee97c91966ff6fcc43 WatchSource:0}: Error finding container 63fa0d2cafad11e02e495970c2280f22c5b4893287e2b1ee97c91966ff6fcc43: Status 404 returned error can't find the container with id 63fa0d2cafad11e02e495970c2280f22c5b4893287e2b1ee97c91966ff6fcc43
Jul 31 23:47:21.399939 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:21.399915    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f44c6f5f8-5hkk9"]
Jul 31 23:47:21.402503 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:21.402479    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode005be7e_fed7_468e_9a7d_601db46f362f.slice/crio-8489ac8c09474b6f287e7ec82d57106ce93201034c1d84d837ff42c3c903ff93 WatchSource:0}: Error finding container 8489ac8c09474b6f287e7ec82d57106ce93201034c1d84d837ff42c3c903ff93: Status 404 returned error can't find the container with id 8489ac8c09474b6f287e7ec82d57106ce93201034c1d84d837ff42c3c903ff93
Jul 31 23:47:21.489707 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:21.489663    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jw8jw" event={"ID":"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd","Type":"ContainerStarted","Data":"e0d2a30d61e6b53ec3ad32aa5dcfb43fa32d1dc04fa7cda2ff6e03dc3cc32e09"}
Jul 31 23:47:21.491191 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:21.491163    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zmcw2" event={"ID":"4eb3a0b2-c573-43c7-a8dd-a01c3545d78b","Type":"ContainerStarted","Data":"98db1ba1e35c867bcdf44817a9db1aa5d8e2faf38bec73613d402bee39b4a3c5"}
Jul 31 23:47:21.493234 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:21.493207    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f44c6f5f8-5hkk9" event={"ID":"e005be7e-fed7-468e-9a7d-601db46f362f","Type":"ContainerStarted","Data":"8489ac8c09474b6f287e7ec82d57106ce93201034c1d84d837ff42c3c903ff93"}
Jul 31 23:47:21.494446 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:21.494421    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw" event={"ID":"81b34b10-cd40-4782-b31e-086b1aa66312","Type":"ContainerStarted","Data":"63fa0d2cafad11e02e495970c2280f22c5b4893287e2b1ee97c91966ff6fcc43"}
Jul 31 23:47:22.501847 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:22.501778    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f44c6f5f8-5hkk9" event={"ID":"e005be7e-fed7-468e-9a7d-601db46f362f","Type":"ContainerStarted","Data":"f634f11d521753781db7ce43d6e74ff41c5af988ea16bd6d879eaafbed14a2ae"}
Jul 31 23:47:22.504323 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:22.504290    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jw8jw" event={"ID":"fec5df48-6c8a-49ff-bb01-43d1d87ca8fd","Type":"ContainerStarted","Data":"eee4d4cdc390202573ff0fc552002792dc935f08a530933969d5cc28a0fc981f"}
Jul 31 23:47:22.504461 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:22.504446    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:47:22.521635 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:22.521390    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zmcw2" podStartSLOduration=67.418419573 podStartE2EDuration="1m12.521372915s" totalImagesPullingTime="5.102953342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 23:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:21.508049473 +0000 UTC m=+71.967512287" watchObservedRunningTime="2026-07-31 23:47:22.521372915 +0000 UTC m=+72.980835733"
Jul 31 23:47:22.521818 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:22.521780    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-f44c6f5f8-5hkk9" podStartSLOduration=35.521769797 podStartE2EDuration="35.521769797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:22.521142787 +0000 UTC m=+72.980605603" watchObservedRunningTime="2026-07-31 23:47:22.521769797 +0000 UTC m=+72.981232612"
Jul 31 23:47:22.539226 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:22.538212    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jw8jw" podStartSLOduration=34.663754609 podStartE2EDuration="38.538195845s" totalImagesPullingTime="3.874441236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:22.538081137 +0000 UTC m=+72.997543949" watchObservedRunningTime="2026-07-31 23:47:22.538195845 +0000 UTC m=+72.997658660"
Jul 31 23:47:23.021508 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:23.021471    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:23.123548 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:23.123377    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:23.192607 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:23.192552    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:23.268275 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:23.268221    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:47:23.271221 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:23.271197    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:47:23.512275 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:23.512215    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw" event={"ID":"81b34b10-cd40-4782-b31e-086b1aa66312","Type":"ContainerStarted","Data":"b45f7c63323b2a644be774f5601b542ba7f78a876e5e31b84fd7d66bf3cd4aad"}
Jul 31 23:47:23.512776 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:23.512746    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:47:23.514123 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:23.514069    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-f44c6f5f8-5hkk9"
Jul 31 23:47:23.547088 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:23.546750    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-c8dsw" podStartSLOduration=34.708943873 podStartE2EDuration="36.546730706s" totalImagesPullingTime="1.837786833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:23.546290901 +0000 UTC m=+74.005753713" watchObservedRunningTime="2026-07-31 23:47:23.546730706 +0000 UTC m=+74.006193519"
Jul 31 23:47:23.598946 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:23.598900    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:23.697631 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:23.697589    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:23.799070 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:23.798921    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:24.178771 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:24.178736    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:24.280315 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:24.280272    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:25.439636 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:25.439588    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:25.882460 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:25.882428    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:47:28.043925 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:28.043893    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:28.145928 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:28.145888    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:28.598776 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:28.598722    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:28.665199 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:28.665165    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:28.793865 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:28.793821    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:28.881169 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:28.881108    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:29.144066 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:29.143975    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:29.237686 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:29.237651    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:30.766318 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:30.766277    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:30.820547 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:30.820479    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:31.374806 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.374766    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:31.432655 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.432613    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:31.764221 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.764148    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cf8958db9-phbrr"]
Jul 31 23:47:31.766393 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.766364    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.768960 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.768935    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 31 23:47:31.770404 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.770054    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ftvcp\""
Jul 31 23:47:31.770404 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.770080    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 31 23:47:31.770404 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.770095    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 31 23:47:31.770404 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.770102    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 31 23:47:31.770404 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.770151    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 31 23:47:31.776807 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.776186    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 31 23:47:31.778032 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.778010    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cf8958db9-phbrr"]
Jul 31 23:47:31.815586 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.815557    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:31.823521 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.823493    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-trusted-ca-bundle\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.823724 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.823533    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-oauth-serving-cert\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.823724 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.823580    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/404cb752-de98-4920-bbf1-59cc2dd03c05-console-serving-cert\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.823724 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.823637    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thh7h\" (UniqueName: \"kubernetes.io/projected/404cb752-de98-4920-bbf1-59cc2dd03c05-kube-api-access-thh7h\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.823724 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.823674    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/404cb752-de98-4920-bbf1-59cc2dd03c05-console-oauth-config\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.823724 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.823702    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-console-config\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.823959 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.823745    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-service-ca\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.925190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.925144    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-trusted-ca-bundle\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.925380 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.925205    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-oauth-serving-cert\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.925380 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.925275    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/404cb752-de98-4920-bbf1-59cc2dd03c05-console-serving-cert\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.925380 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.925312    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thh7h\" (UniqueName: \"kubernetes.io/projected/404cb752-de98-4920-bbf1-59cc2dd03c05-kube-api-access-thh7h\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.925380 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.925351    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/404cb752-de98-4920-bbf1-59cc2dd03c05-console-oauth-config\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.925380 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.925379    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-console-config\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.925642 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.925399    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-service-ca\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.926113 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.926080    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-oauth-serving-cert\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.926113 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.926098    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-console-config\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.926329 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.926145    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-trusted-ca-bundle\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.926329 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.926145    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-service-ca\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.928133 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.928089    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/404cb752-de98-4920-bbf1-59cc2dd03c05-console-serving-cert\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.928458 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.928427    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/404cb752-de98-4920-bbf1-59cc2dd03c05-console-oauth-config\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:31.934183 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:31.934147    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thh7h\" (UniqueName: \"kubernetes.io/projected/404cb752-de98-4920-bbf1-59cc2dd03c05-kube-api-access-thh7h\") pod \"console-7cf8958db9-phbrr\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") " pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:32.078809 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:32.078731    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:32.405448 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:32.405411    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:32.502834 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:32.502801    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:32.514649 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:32.514622    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jw8jw"
Jul 31 23:47:33.546613 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:33.546588    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cf8958db9-phbrr"]
Jul 31 23:47:33.550102 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:33.550078    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404cb752_de98_4920_bbf1_59cc2dd03c05.slice/crio-320fa8faa1ada038540e853d5a9624c18e62a4fc6f6bb56c81c0519690990c54 WatchSource:0}: Error finding container 320fa8faa1ada038540e853d5a9624c18e62a4fc6f6bb56c81c0519690990c54: Status 404 returned error can't find the container with id 320fa8faa1ada038540e853d5a9624c18e62a4fc6f6bb56c81c0519690990c54
Jul 31 23:47:33.997502 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:33.997412    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:34.194658 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.194624    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:34.282623 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.282051    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bv2b6"]
Jul 31 23:47:34.285450 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.285429    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.289803 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.289325    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7cndp\""
Jul 31 23:47:34.289803 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.289569    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 23:47:34.290213 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.290190    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 23:47:34.290729 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.290430    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 23:47:34.290729 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.290447    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 23:47:34.327989 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.327957    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:34.344455 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.344428    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35b7e250-b205-4ec0-aa2a-32b283a15e8b-metrics-client-ca\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.344583 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.344487    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35b7e250-b205-4ec0-aa2a-32b283a15e8b-node-exporter-textfile\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.344583 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.344560    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35b7e250-b205-4ec0-aa2a-32b283a15e8b-root\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.344706 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.344597    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35b7e250-b205-4ec0-aa2a-32b283a15e8b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.344706 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.344636    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjfp\" (UniqueName: \"kubernetes.io/projected/35b7e250-b205-4ec0-aa2a-32b283a15e8b-kube-api-access-pqjfp\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.344706 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.344681    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35b7e250-b205-4ec0-aa2a-32b283a15e8b-sys\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.344857 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.344713    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35b7e250-b205-4ec0-aa2a-32b283a15e8b-node-exporter-tls\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.344857 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.344739    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35b7e250-b205-4ec0-aa2a-32b283a15e8b-node-exporter-wtmp\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.344857 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.344804    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35b7e250-b205-4ec0-aa2a-32b283a15e8b-node-exporter-accelerators-collector-config\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.446181 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.446141    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35b7e250-b205-4ec0-aa2a-32b283a15e8b-root\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.446378 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.446190    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35b7e250-b205-4ec0-aa2a-32b283a15e8b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.446378 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.446234    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjfp\" (UniqueName: \"kubernetes.io/projected/35b7e250-b205-4ec0-aa2a-32b283a15e8b-kube-api-access-pqjfp\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.446378 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.446334    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35b7e250-b205-4ec0-aa2a-32b283a15e8b-sys\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.446564 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.446416    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35b7e250-b205-4ec0-aa2a-32b283a15e8b-node-exporter-tls\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.446564 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.446441    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35b7e250-b205-4ec0-aa2a-32b283a15e8b-node-exporter-wtmp\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.446564 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.446479    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35b7e250-b205-4ec0-aa2a-32b283a15e8b-node-exporter-accelerators-collector-config\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.446564 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.446517    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35b7e250-b205-4ec0-aa2a-32b283a15e8b-metrics-client-ca\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.446564 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.446550    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/35b7e250-b205-4ec0-aa2a-32b283a15e8b-root\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.446826 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.446605    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35b7e250-b205-4ec0-aa2a-32b283a15e8b-node-exporter-textfile\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.446826 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.446790    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35b7e250-b205-4ec0-aa2a-32b283a15e8b-sys\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.446937 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.446907    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/35b7e250-b205-4ec0-aa2a-32b283a15e8b-node-exporter-textfile\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.447353 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.447314    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/35b7e250-b205-4ec0-aa2a-32b283a15e8b-node-exporter-accelerators-collector-config\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.447353 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.447323    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/35b7e250-b205-4ec0-aa2a-32b283a15e8b-node-exporter-wtmp\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.447525 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.447427    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35b7e250-b205-4ec0-aa2a-32b283a15e8b-metrics-client-ca\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.454127 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.452212    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/35b7e250-b205-4ec0-aa2a-32b283a15e8b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.457734 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.457579    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:34.459110 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.456348    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/35b7e250-b205-4ec0-aa2a-32b283a15e8b-node-exporter-tls\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.476367 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.476338    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjfp\" (UniqueName: \"kubernetes.io/projected/35b7e250-b205-4ec0-aa2a-32b283a15e8b-kube-api-access-pqjfp\") pod \"node-exporter-bv2b6\" (UID: \"35b7e250-b205-4ec0-aa2a-32b283a15e8b\") " pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.555600 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.555482    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:34.559344 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.559201    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf8958db9-phbrr" event={"ID":"404cb752-de98-4920-bbf1-59cc2dd03c05","Type":"ContainerStarted","Data":"320fa8faa1ada038540e853d5a9624c18e62a4fc6f6bb56c81c0519690990c54"}
Jul 31 23:47:34.562181 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.562152    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-wl8v7" event={"ID":"7ed00006-4005-410f-b5f0-cc72951ee8b4","Type":"ContainerStarted","Data":"48e28b494b8863bcd0a68d775f56b57d0e2fd8a694b6eb088cefde456de9940b"}
Jul 31 23:47:34.562880 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.562817    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-wl8v7"
Jul 31 23:47:34.579715 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.579688    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-wl8v7"
Jul 31 23:47:34.585885 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.585728    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-wl8v7" podStartSLOduration=2.543554351 podStartE2EDuration="19.585708592s" totalImagesPullingTime="17.042154241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:34.584626599 +0000 UTC m=+85.044089414" watchObservedRunningTime="2026-07-31 23:47:34.585708592 +0000 UTC m=+85.045171408"
Jul 31 23:47:34.601048 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.600976    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bv2b6"
Jul 31 23:47:34.640067 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:34.639130    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b7e250_b205_4ec0_aa2a_32b283a15e8b.slice/crio-e21c37388d126fa0592a2c0c42b40ae056fd6d2c873984972b9f02db83482cc7 WatchSource:0}: Error finding container e21c37388d126fa0592a2c0c42b40ae056fd6d2c873984972b9f02db83482cc7: Status 404 returned error can't find the container with id e21c37388d126fa0592a2c0c42b40ae056fd6d2c873984972b9f02db83482cc7
Jul 31 23:47:34.705838 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:34.705798    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:35.205648 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.205598    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:35.296637 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.296585    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:35.301189 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.301029    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:47:35.327809 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.326603    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.327809 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.327050    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:47:35.330304 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.330197    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 31 23:47:35.330531 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.330512    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 31 23:47:35.331075 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.331045    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 31 23:47:35.331377 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.331358    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 31 23:47:35.333295 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.331658    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 31 23:47:35.333295 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.331889    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 31 23:47:35.333295 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.331923    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 31 23:47:35.333295 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.331975    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 31 23:47:35.336530 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.335038    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 31 23:47:35.336530 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.335272    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ppj72\""
Jul 31 23:47:35.356289 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.355573    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.356289 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.355614    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.356289 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.355646    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.356289 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.355676    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.356289 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.355709    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.356289 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.355740    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.356289 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.355861    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.356289 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.355887    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-config-out\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.356289 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.355931    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-web-config\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.356289 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.355954    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddtsc\" (UniqueName: \"kubernetes.io/projected/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-kube-api-access-ddtsc\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.356289 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.355986    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.356289 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.356011    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.356289 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.356073    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.457408 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.457164    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.457408 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.457220    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.457408 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.457272    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.457408 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.457303    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.460547 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:47:35.458178    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-alertmanager-trusted-ca-bundle podName:c69b96d7-838c-4c38-88e8-c4ab90f0d2f3 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:35.958054707 +0000 UTC m=+86.417517499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3") : configmap references non-existent config key: ca-bundle.crt
Jul 31 23:47:35.460547 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.460229    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.462441 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.461109    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.462441 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.461160    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.462441 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.461190    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-config-out\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.462441 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.461232    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-web-config\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.462441 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.461273    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddtsc\" (UniqueName: \"kubernetes.io/projected/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-kube-api-access-ddtsc\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.462441 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.461304    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.462441 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.461332    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.462441 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.461396    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.462441 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.461436    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.462441 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.461462    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.462441 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:47:35.461595    2546 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 31 23:47:35.462441 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:47:35.461643    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-main-tls podName:c69b96d7-838c-4c38-88e8-c4ab90f0d2f3 nodeName:}" failed. No retries permitted until 2026-07-31 23:47:35.961626378 +0000 UTC m=+86.421089194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3") : secret "alertmanager-main-tls" not found
Jul 31 23:47:35.463071 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.462640    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.467106 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.467054    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.468548 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.468500    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.468646 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.468560    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-web-config\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.471917 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.471874    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddtsc\" (UniqueName: \"kubernetes.io/projected/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-kube-api-access-ddtsc\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.474234 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.474186    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.474347 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.474245    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.477104 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.477065    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.489375 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.489333    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-config-out\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.510786 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.510740    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:35.574442 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.574369    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bv2b6" event={"ID":"35b7e250-b205-4ec0-aa2a-32b283a15e8b","Type":"ContainerStarted","Data":"e21c37388d126fa0592a2c0c42b40ae056fd6d2c873984972b9f02db83482cc7"}
Jul 31 23:47:35.613243 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.613191    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:35.757514 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.757414    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:35.846529 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.846429    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:35.943853 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.943820    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:35.966714 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.966687    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.966878 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.966743    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.967582 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.967555    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:35.970155 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:35.970129    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:36.240805 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:36.240773    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:47:37.156704 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.156640    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:37.260214 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.260132    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:37.318787 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.318568    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-86669fc994-n7lns"]
Jul 31 23:47:37.336186 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.336058    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:37.357764 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.357727    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86669fc994-n7lns"]
Jul 31 23:47:37.357916 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.357868    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.361798 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.361665    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 31 23:47:37.361798 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.361706    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 31 23:47:37.362288 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.361883    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 31 23:47:37.362288 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.361930    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 31 23:47:37.362288 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.362200    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-45tiqa4krq7i4\""
Jul 31 23:47:37.362507 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.362430    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 31 23:47:37.362692 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.362677    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gsqmd\""
Jul 31 23:47:37.473705 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.473642    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65d74d959b-7g6nj"
Jul 31 23:47:37.483636 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.483609    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9phk2\" (UniqueName: \"kubernetes.io/projected/81065d77-f670-4ddc-a06a-04a88c98fc8e-kube-api-access-9phk2\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.483773 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.483652    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.483773 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.483688    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-thanos-querier-tls\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.483773 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.483707    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.483773 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.483739    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81065d77-f670-4ddc-a06a-04a88c98fc8e-metrics-client-ca\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.483773 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.483759    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-grpc-tls\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.483974 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.483863    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.483974 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.483917    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.585153 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.585121    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9phk2\" (UniqueName: \"kubernetes.io/projected/81065d77-f670-4ddc-a06a-04a88c98fc8e-kube-api-access-9phk2\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.585300 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.585199    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.585300 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.585266    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-thanos-querier-tls\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.585300 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.585290    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.585472 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.585341    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81065d77-f670-4ddc-a06a-04a88c98fc8e-metrics-client-ca\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.585472 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.585374    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-grpc-tls\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.585558 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.585489    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.585558 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.585538    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.586441 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.586413    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81065d77-f670-4ddc-a06a-04a88c98fc8e-metrics-client-ca\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.588553 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.588529    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.589404 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.589368    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:37.590207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.590184    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-grpc-tls\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.590325 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.590188    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-thanos-querier-tls\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.590612 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.590588    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.590802 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.590570    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.592516 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.592494    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81065d77-f670-4ddc-a06a-04a88c98fc8e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.593897 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.593878    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9phk2\" (UniqueName: \"kubernetes.io/projected/81065d77-f670-4ddc-a06a-04a88c98fc8e-kube-api-access-9phk2\") pod \"thanos-querier-86669fc994-n7lns\" (UID: \"81065d77-f670-4ddc-a06a-04a88c98fc8e\") " pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.668823 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.668794    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:37.690423 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.690393    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:37.790861 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.790781    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:37.890453 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.890336    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:37.940777 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.940117    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:47:37.943520 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:37.943489    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69b96d7_838c_4c38_88e8_c4ab90f0d2f3.slice/crio-dc29fa637b7704d933f96fb3ba3b8976a40bb565af7cc234155128b19136f061 WatchSource:0}: Error finding container dc29fa637b7704d933f96fb3ba3b8976a40bb565af7cc234155128b19136f061: Status 404 returned error can't find the container with id dc29fa637b7704d933f96fb3ba3b8976a40bb565af7cc234155128b19136f061
Jul 31 23:47:37.963048 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.962967    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86669fc994-n7lns"]
Jul 31 23:47:37.968003 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:37.967975    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81065d77_f670_4ddc_a06a_04a88c98fc8e.slice/crio-ae9a8ba73f6be0715a138816e0c54e4386ead0404a30d0531417509f1859c6ef WatchSource:0}: Error finding container ae9a8ba73f6be0715a138816e0c54e4386ead0404a30d0531417509f1859c6ef: Status 404 returned error can't find the container with id ae9a8ba73f6be0715a138816e0c54e4386ead0404a30d0531417509f1859c6ef
Jul 31 23:47:37.970391 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:37.970355    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:38.594456 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.593955    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf8958db9-phbrr" event={"ID":"404cb752-de98-4920-bbf1-59cc2dd03c05","Type":"ContainerStarted","Data":"319f17ae83eeeaa565cf4eec3dbf6538db17dc2910f89b162eba36e0e35a73b8"}
Jul 31 23:47:38.596473 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.596363    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3","Type":"ContainerStarted","Data":"dc29fa637b7704d933f96fb3ba3b8976a40bb565af7cc234155128b19136f061"}
Jul 31 23:47:38.598994 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.598966    2546 generic.go:356] "Generic (PLEG): container finished" podID="35b7e250-b205-4ec0-aa2a-32b283a15e8b" containerID="509aa8c87c38ebff9222610409ce1610d3476ecd5231b2a1d51f2a4b226020b6" exitCode=0
Jul 31 23:47:38.599105 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.599084    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bv2b6" event={"ID":"35b7e250-b205-4ec0-aa2a-32b283a15e8b","Type":"ContainerDied","Data":"509aa8c87c38ebff9222610409ce1610d3476ecd5231b2a1d51f2a4b226020b6"}
Jul 31 23:47:38.602552 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.602517    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86669fc994-n7lns" event={"ID":"81065d77-f670-4ddc-a06a-04a88c98fc8e","Type":"ContainerStarted","Data":"ae9a8ba73f6be0715a138816e0c54e4386ead0404a30d0531417509f1859c6ef"}
Jul 31 23:47:38.617654 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.617601    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cf8958db9-phbrr" podStartSLOduration=3.405409661 podStartE2EDuration="7.617568771s" totalImagesPullingTime="4.21215911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:38.615381831 +0000 UTC m=+89.074844663" watchObservedRunningTime="2026-07-31 23:47:38.617568771 +0000 UTC m=+89.077031584"
Jul 31 23:47:38.739281 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.739154    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6dc885d577-728pn"]
Jul 31 23:47:38.753476 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.752293    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.755285 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.755187    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dc885d577-728pn"]
Jul 31 23:47:38.757156 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.756327    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 31 23:47:38.757156 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.756621    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zgbfv\""
Jul 31 23:47:38.757156 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.756830    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 31 23:47:38.757156 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.757064    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3odhvjhasbq6i\""
Jul 31 23:47:38.759697 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.757959    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 31 23:47:38.759697 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.757995    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 31 23:47:38.802170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.797229    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d286d217-c582-4af1-814c-efe3f56cf7f5-client-ca-bundle\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.802170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.797310    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d286d217-c582-4af1-814c-efe3f56cf7f5-metrics-server-audit-profiles\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.802170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.797427    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d286d217-c582-4af1-814c-efe3f56cf7f5-audit-log\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.802170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.797482    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d286d217-c582-4af1-814c-efe3f56cf7f5-secret-metrics-server-client-certs\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.802170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.797511    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d286d217-c582-4af1-814c-efe3f56cf7f5-secret-metrics-server-tls\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.802170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.797543    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d286d217-c582-4af1-814c-efe3f56cf7f5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.802170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.797573    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcfhq\" (UniqueName: \"kubernetes.io/projected/d286d217-c582-4af1-814c-efe3f56cf7f5-kube-api-access-rcfhq\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.899321 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.899289    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d286d217-c582-4af1-814c-efe3f56cf7f5-client-ca-bundle\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.899497 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.899349    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d286d217-c582-4af1-814c-efe3f56cf7f5-metrics-server-audit-profiles\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.899652 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.899628    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d286d217-c582-4af1-814c-efe3f56cf7f5-audit-log\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.899716 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.899701    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d286d217-c582-4af1-814c-efe3f56cf7f5-secret-metrics-server-client-certs\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.899771 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.899713    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d286d217-c582-4af1-814c-efe3f56cf7f5-audit-log\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.899771 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.899731    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d286d217-c582-4af1-814c-efe3f56cf7f5-secret-metrics-server-tls\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.899872 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.899781    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d286d217-c582-4af1-814c-efe3f56cf7f5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.899872 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.899834    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcfhq\" (UniqueName: \"kubernetes.io/projected/d286d217-c582-4af1-814c-efe3f56cf7f5-kube-api-access-rcfhq\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.901198 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.901141    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d286d217-c582-4af1-814c-efe3f56cf7f5-metrics-server-audit-profiles\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.901324 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.901239    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d286d217-c582-4af1-814c-efe3f56cf7f5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.904836 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.903422    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d286d217-c582-4af1-814c-efe3f56cf7f5-client-ca-bundle\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.904836 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.903885    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d286d217-c582-4af1-814c-efe3f56cf7f5-secret-metrics-server-client-certs\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.905473 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.905441    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d286d217-c582-4af1-814c-efe3f56cf7f5-secret-metrics-server-tls\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.912685 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.912647    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcfhq\" (UniqueName: \"kubernetes.io/projected/d286d217-c582-4af1-814c-efe3f56cf7f5-kube-api-access-rcfhq\") pod \"metrics-server-6dc885d577-728pn\" (UID: \"d286d217-c582-4af1-814c-efe3f56cf7f5\") " pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:38.985382 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:38.985206    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:39.068706 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:39.068523    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:39.069403 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:39.069357    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:39.298155 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:39.298074    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:39.565517 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:39.565443    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dc885d577-728pn"]
Jul 31 23:47:39.610672 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:39.610637    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bv2b6" event={"ID":"35b7e250-b205-4ec0-aa2a-32b283a15e8b","Type":"ContainerStarted","Data":"eadfae04c7ba50f59c0238a99c279b0d07a2cca2037752a83cceb4b517449325"}
Jul 31 23:47:39.611129 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:39.610708    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bv2b6" event={"ID":"35b7e250-b205-4ec0-aa2a-32b283a15e8b","Type":"ContainerStarted","Data":"d59d0f90def61ddae56d979ab4ce27bfeca3be1415f97b7aa0a7da418eda44e5"}
Jul 31 23:47:39.636663 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:39.636610    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bv2b6" podStartSLOduration=2.4254254729999998 podStartE2EDuration="5.636590435s" totalImagesPullingTime="3.110746758s" totalInitContainerRuntime="100.418204ms" isStatefulPod=true podCreationTimestamp="2026-07-31 23:47:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:39.635949735 +0000 UTC m=+90.095412552" watchObservedRunningTime="2026-07-31 23:47:39.636590435 +0000 UTC m=+90.096053249"
Jul 31 23:47:40.313010 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:40.312978    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd286d217_c582_4af1_814c_efe3f56cf7f5.slice/crio-50757ac959d6afd5baa24912422b07d44d4bd283815287f59976c127d6c76e4a WatchSource:0}: Error finding container 50757ac959d6afd5baa24912422b07d44d4bd283815287f59976c127d6c76e4a: Status 404 returned error can't find the container with id 50757ac959d6afd5baa24912422b07d44d4bd283815287f59976c127d6c76e4a
Jul 31 23:47:40.502972 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:40.502933    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:40.615386 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:40.615351    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dc885d577-728pn" event={"ID":"d286d217-c582-4af1-814c-efe3f56cf7f5","Type":"ContainerStarted","Data":"50757ac959d6afd5baa24912422b07d44d4bd283815287f59976c127d6c76e4a"}
Jul 31 23:47:40.893152 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:40.893117    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx" podUID="59605a10-07f0-4484-bdda-7779bc9c8316" containerName="registry" containerID="cri-o://af9fe373b0a67c2535b9600e0977260c72e6a7d90ea1d5e90ebd58cee757787b" gracePeriod=30
Jul 31 23:47:41.185167 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.185137    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:47:41.320222 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.320151    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-bound-sa-token\") pod \"59605a10-07f0-4484-bdda-7779bc9c8316\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") "
Jul 31 23:47:41.320222 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.320196    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59605a10-07f0-4484-bdda-7779bc9c8316-installation-pull-secrets\") pod \"59605a10-07f0-4484-bdda-7779bc9c8316\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") "
Jul 31 23:47:41.320222 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.320229    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls\") pod \"59605a10-07f0-4484-bdda-7779bc9c8316\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") "
Jul 31 23:47:41.320506 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.320278    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59605a10-07f0-4484-bdda-7779bc9c8316-image-registry-private-configuration\") pod \"59605a10-07f0-4484-bdda-7779bc9c8316\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") "
Jul 31 23:47:41.321278 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.320617    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59605a10-07f0-4484-bdda-7779bc9c8316-trusted-ca\") pod \"59605a10-07f0-4484-bdda-7779bc9c8316\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") "
Jul 31 23:47:41.321278 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.320694    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkgw4\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-kube-api-access-xkgw4\") pod \"59605a10-07f0-4484-bdda-7779bc9c8316\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") "
Jul 31 23:47:41.321278 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.320739    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59605a10-07f0-4484-bdda-7779bc9c8316-ca-trust-extracted\") pod \"59605a10-07f0-4484-bdda-7779bc9c8316\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") "
Jul 31 23:47:41.321278 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.320805    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59605a10-07f0-4484-bdda-7779bc9c8316-registry-certificates\") pod \"59605a10-07f0-4484-bdda-7779bc9c8316\" (UID: \"59605a10-07f0-4484-bdda-7779bc9c8316\") "
Jul 31 23:47:41.321567 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.321274    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59605a10-07f0-4484-bdda-7779bc9c8316-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "59605a10-07f0-4484-bdda-7779bc9c8316" (UID: "59605a10-07f0-4484-bdda-7779bc9c8316"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:41.322104 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.322075    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59605a10-07f0-4484-bdda-7779bc9c8316-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "59605a10-07f0-4484-bdda-7779bc9c8316" (UID: "59605a10-07f0-4484-bdda-7779bc9c8316"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:47:41.324565 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.324524    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-kube-api-access-xkgw4" (OuterVolumeSpecName: "kube-api-access-xkgw4") pod "59605a10-07f0-4484-bdda-7779bc9c8316" (UID: "59605a10-07f0-4484-bdda-7779bc9c8316"). InnerVolumeSpecName "kube-api-access-xkgw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:47:41.328798 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.328770    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59605a10-07f0-4484-bdda-7779bc9c8316-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "59605a10-07f0-4484-bdda-7779bc9c8316" (UID: "59605a10-07f0-4484-bdda-7779bc9c8316"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:47:41.329027 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.328988    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59605a10-07f0-4484-bdda-7779bc9c8316-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "59605a10-07f0-4484-bdda-7779bc9c8316" (UID: "59605a10-07f0-4484-bdda-7779bc9c8316"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:47:41.329194 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.329103    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "59605a10-07f0-4484-bdda-7779bc9c8316" (UID: "59605a10-07f0-4484-bdda-7779bc9c8316"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:47:41.330488 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.330457    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "59605a10-07f0-4484-bdda-7779bc9c8316" (UID: "59605a10-07f0-4484-bdda-7779bc9c8316"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:47:41.333088 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.333054    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59605a10-07f0-4484-bdda-7779bc9c8316-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "59605a10-07f0-4484-bdda-7779bc9c8316" (UID: "59605a10-07f0-4484-bdda-7779bc9c8316"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:47:41.422429 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.422395    2546 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/59605a10-07f0-4484-bdda-7779bc9c8316-ca-trust-extracted\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:47:41.422429 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.422428    2546 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/59605a10-07f0-4484-bdda-7779bc9c8316-registry-certificates\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:47:41.422676 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.422445    2546 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-bound-sa-token\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:47:41.422676 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.422461    2546 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/59605a10-07f0-4484-bdda-7779bc9c8316-installation-pull-secrets\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:47:41.422676 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.422476    2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-registry-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:47:41.422676 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.422492    2546 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/59605a10-07f0-4484-bdda-7779bc9c8316-image-registry-private-configuration\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:47:41.422676 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.422507    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/59605a10-07f0-4484-bdda-7779bc9c8316-trusted-ca\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:47:41.422676 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.422521    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkgw4\" (UniqueName: \"kubernetes.io/projected/59605a10-07f0-4484-bdda-7779bc9c8316-kube-api-access-xkgw4\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:47:41.624211 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.624170    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86669fc994-n7lns" event={"ID":"81065d77-f670-4ddc-a06a-04a88c98fc8e","Type":"ContainerStarted","Data":"8f27150973270f62ec7ac5605b7afe62ed89fe08a6371f9957f10dcf9043689a"}
Jul 31 23:47:41.624611 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.624281    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86669fc994-n7lns" event={"ID":"81065d77-f670-4ddc-a06a-04a88c98fc8e","Type":"ContainerStarted","Data":"9d29a31de2d9d00c08dc2534f3c537280b56a2312bf230436825b412e880685e"}
Jul 31 23:47:41.624611 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.624327    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86669fc994-n7lns" event={"ID":"81065d77-f670-4ddc-a06a-04a88c98fc8e","Type":"ContainerStarted","Data":"224a647cfda2ddb9c02f3545c2baef7b9e39e392b5f8fcdd55f2be4191354a90"}
Jul 31 23:47:41.626662 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.626631    2546 generic.go:356] "Generic (PLEG): container finished" podID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerID="de5ca77dee7bf355a5e1ee6a94cc514bf2301f06f2d532307ca2e94745673efa" exitCode=0
Jul 31 23:47:41.626847 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.626824    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3","Type":"ContainerDied","Data":"de5ca77dee7bf355a5e1ee6a94cc514bf2301f06f2d532307ca2e94745673efa"}
Jul 31 23:47:41.629504 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.629478    2546 generic.go:356] "Generic (PLEG): container finished" podID="59605a10-07f0-4484-bdda-7779bc9c8316" containerID="af9fe373b0a67c2535b9600e0977260c72e6a7d90ea1d5e90ebd58cee757787b" exitCode=0
Jul 31 23:47:41.629601 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.629535    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx" event={"ID":"59605a10-07f0-4484-bdda-7779bc9c8316","Type":"ContainerDied","Data":"af9fe373b0a67c2535b9600e0977260c72e6a7d90ea1d5e90ebd58cee757787b"}
Jul 31 23:47:41.629601 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.629572    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx" event={"ID":"59605a10-07f0-4484-bdda-7779bc9c8316","Type":"ContainerDied","Data":"57f5bfe6d6f9599f17c0c2dd4b6d79c7daaeb43b0003e508e73b7e4af2c9a959"}
Jul 31 23:47:41.629713 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.629599    2546 scope.go:117] "RemoveContainer" containerID="af9fe373b0a67c2535b9600e0977260c72e6a7d90ea1d5e90ebd58cee757787b"
Jul 31 23:47:41.629767 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.629746    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6b94dfb9-cngmx"
Jul 31 23:47:41.670942 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.670833    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f6b94dfb9-cngmx"]
Jul 31 23:47:41.677449 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:41.677393    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f6b94dfb9-cngmx"]
Jul 31 23:47:42.077567 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:42.077493    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59605a10-07f0-4484-bdda-7779bc9c8316" path="/var/lib/kubelet/pods/59605a10-07f0-4484-bdda-7779bc9c8316/volumes"
Jul 31 23:47:42.079485 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:42.079453    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:42.079626 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:42.079525    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:42.085160 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:42.085137    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:42.639035 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:42.639007    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:47:42.710775 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:42.710703    2546 scope.go:117] "RemoveContainer" containerID="af9fe373b0a67c2535b9600e0977260c72e6a7d90ea1d5e90ebd58cee757787b"
Jul 31 23:47:42.711142 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:47:42.711105    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9fe373b0a67c2535b9600e0977260c72e6a7d90ea1d5e90ebd58cee757787b\": container with ID starting with af9fe373b0a67c2535b9600e0977260c72e6a7d90ea1d5e90ebd58cee757787b not found: ID does not exist" containerID="af9fe373b0a67c2535b9600e0977260c72e6a7d90ea1d5e90ebd58cee757787b"
Jul 31 23:47:42.711235 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:42.711150    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9fe373b0a67c2535b9600e0977260c72e6a7d90ea1d5e90ebd58cee757787b"} err="failed to get container status \"af9fe373b0a67c2535b9600e0977260c72e6a7d90ea1d5e90ebd58cee757787b\": rpc error: code = NotFound desc = could not find container \"af9fe373b0a67c2535b9600e0977260c72e6a7d90ea1d5e90ebd58cee757787b\": container with ID starting with af9fe373b0a67c2535b9600e0977260c72e6a7d90ea1d5e90ebd58cee757787b not found: ID does not exist"
Jul 31 23:47:42.824000 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:42.823952    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:42.925911 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:42.925884    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:43.216577 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.216511    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bdc56cf6b-598vg"]
Jul 31 23:47:43.216949 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.216935    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59605a10-07f0-4484-bdda-7779bc9c8316" containerName="registry"
Jul 31 23:47:43.216996 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.216952    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="59605a10-07f0-4484-bdda-7779bc9c8316" containerName="registry"
Jul 31 23:47:43.217047 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.217037    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="59605a10-07f0-4484-bdda-7779bc9c8316" containerName="registry"
Jul 31 23:47:43.232938 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.232887    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.233401 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.233379    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bdc56cf6b-598vg"]
Jul 31 23:47:43.340049 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.340017    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffd8234c-196e-437d-922a-8452c780c3aa-console-oauth-config\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.340226 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.340067    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-service-ca\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.340226 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.340100    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-console-config\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.340226 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.340158    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffd8234c-196e-437d-922a-8452c780c3aa-console-serving-cert\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.340226 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.340210    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-trusted-ca-bundle\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.340421 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.340229    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-oauth-serving-cert\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.340421 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.340316    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rf26\" (UniqueName: \"kubernetes.io/projected/ffd8234c-196e-437d-922a-8452c780c3aa-kube-api-access-7rf26\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.440773 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.440736    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rf26\" (UniqueName: \"kubernetes.io/projected/ffd8234c-196e-437d-922a-8452c780c3aa-kube-api-access-7rf26\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.440931 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.440792    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffd8234c-196e-437d-922a-8452c780c3aa-console-oauth-config\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.440931 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.440837    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-service-ca\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.440931 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.440871    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-console-config\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.440931 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.440903    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffd8234c-196e-437d-922a-8452c780c3aa-console-serving-cert\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.441146 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.440959    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-trusted-ca-bundle\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.441146 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.440983    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-oauth-serving-cert\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.442814 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.441821    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-oauth-serving-cert\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.442814 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.442025    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-console-config\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.442814 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.442188    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-service-ca\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.442814 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.442774    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-trusted-ca-bundle\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.444444 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.444383    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffd8234c-196e-437d-922a-8452c780c3aa-console-oauth-config\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.444881 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.444854    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffd8234c-196e-437d-922a-8452c780c3aa-console-serving-cert\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.460868 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.460309    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rf26\" (UniqueName: \"kubernetes.io/projected/ffd8234c-196e-437d-922a-8452c780c3aa-kube-api-access-7rf26\") pod \"console-bdc56cf6b-598vg\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") " pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.545248 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.545171    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:43.644207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.644173    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86669fc994-n7lns" event={"ID":"81065d77-f670-4ddc-a06a-04a88c98fc8e","Type":"ContainerStarted","Data":"eb84493370d8abcfaa2c5d961012bcacd3e75fce1420c695bf26c0618ccef4b2"}
Jul 31 23:47:43.644644 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.644243    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86669fc994-n7lns" event={"ID":"81065d77-f670-4ddc-a06a-04a88c98fc8e","Type":"ContainerStarted","Data":"35268ce7ac26e57525e4121c0b1d2851aa22ac6de3cea2ddc4de08593cbca5d1"}
Jul 31 23:47:43.644644 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.644313    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:43.644644 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.644356    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86669fc994-n7lns" event={"ID":"81065d77-f670-4ddc-a06a-04a88c98fc8e","Type":"ContainerStarted","Data":"445c2afe22963d067bec4d64f6a00860256e9320c54adf32b85df165b1f286a1"}
Jul 31 23:47:43.651368 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.651272    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dc885d577-728pn" event={"ID":"d286d217-c582-4af1-814c-efe3f56cf7f5","Type":"ContainerStarted","Data":"b32a8d39e964879d8934483c48572b42d0571c3e4a6c8c3cb5c8a5d866e69508"}
Jul 31 23:47:43.682954 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.682915    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-86669fc994-n7lns" podStartSLOduration=2.348964563 podStartE2EDuration="6.682901855s" totalImagesPullingTime="4.333937292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:43.682331657 +0000 UTC m=+94.141794473" watchObservedRunningTime="2026-07-31 23:47:43.682901855 +0000 UTC m=+94.142364668"
Jul 31 23:47:43.735770 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.735716    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6dc885d577-728pn" podStartSLOduration=3.513267719 podStartE2EDuration="5.735699407s" totalImagesPullingTime="2.222431688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:43.735675779 +0000 UTC m=+94.195138592" watchObservedRunningTime="2026-07-31 23:47:43.735699407 +0000 UTC m=+94.195162220"
Jul 31 23:47:43.745967 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:43.745941    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bdc56cf6b-598vg"]
Jul 31 23:47:43.748907 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:47:43.748882    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd8234c_196e_437d_922a_8452c780c3aa.slice/crio-c00b4f472ce7fd427bc5e08a92f1c21b0d81ca71ead9d5255c9381e8a154d1df WatchSource:0}: Error finding container c00b4f472ce7fd427bc5e08a92f1c21b0d81ca71ead9d5255c9381e8a154d1df: Status 404 returned error can't find the container with id c00b4f472ce7fd427bc5e08a92f1c21b0d81ca71ead9d5255c9381e8a154d1df
Jul 31 23:47:44.188178 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:44.188149    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:44.285229 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:44.285202    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:44.657746 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:44.657716    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bdc56cf6b-598vg" event={"ID":"ffd8234c-196e-437d-922a-8452c780c3aa","Type":"ContainerStarted","Data":"54de3cd581547cd11d920a16c5f3adcb7aa7174adf731067e7d3a5c061d3bd43"}
Jul 31 23:47:44.658061 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:44.657773    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bdc56cf6b-598vg" event={"ID":"ffd8234c-196e-437d-922a-8452c780c3aa","Type":"ContainerStarted","Data":"c00b4f472ce7fd427bc5e08a92f1c21b0d81ca71ead9d5255c9381e8a154d1df"}
Jul 31 23:47:44.661191 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:44.661165    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3","Type":"ContainerStarted","Data":"13f7fe4aad4908450b840e53b5314ea7d28bc43a784e4571c7645bd87b33cf46"}
Jul 31 23:47:44.661317 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:44.661243    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3","Type":"ContainerStarted","Data":"9b7a7b360b42b4deb975ff78cfb33fa86b80704d0420cb3ebb083769005177f0"}
Jul 31 23:47:44.661317 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:44.661307    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3","Type":"ContainerStarted","Data":"6944bac4cf2013b79c14f315c783dcfb4a07bc2881258ee50f69da9ba911882b"}
Jul 31 23:47:44.661428 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:44.661351    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3","Type":"ContainerStarted","Data":"8bc5a5884d7cf7d3f0aa81b8e2a6e27c5d370e5d1ee4c4553d0a4e8a2cf3a34a"}
Jul 31 23:47:44.661428 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:44.661396    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3","Type":"ContainerStarted","Data":"5d5331dd13a9bdb70bf698fcd5a546cb776bea29b27853ebb96e5d2581cfa07f"}
Jul 31 23:47:44.678362 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:44.678321    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bdc56cf6b-598vg" podStartSLOduration=1.6783068079999999 podStartE2EDuration="1.678306808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:44.677371035 +0000 UTC m=+95.136833848" watchObservedRunningTime="2026-07-31 23:47:44.678306808 +0000 UTC m=+95.137769621"
Jul 31 23:47:45.668416 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:45.668380    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3","Type":"ContainerStarted","Data":"c72d84957629906ed1b2770083009292292e0ff7ec20456780c52942957855d1"}
Jul 31 23:47:45.699397 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:45.699342    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.669468114 podStartE2EDuration="10.699320932s" totalImagesPullingTime="4.999091491s" totalInitContainerRuntime="30.761327ms" isStatefulPod=false podCreationTimestamp="2026-07-31 23:47:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:47:45.696360163 +0000 UTC m=+96.155822977" watchObservedRunningTime="2026-07-31 23:47:45.699320932 +0000 UTC m=+96.158783746"
Jul 31 23:47:46.086581 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:46.086495    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:46.189036 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:46.189000    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:47.471172 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:47.471142    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wr59z"
Jul 31 23:47:49.417391 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:49.417357    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:49.518829 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:49.518792    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:49.578062 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:49.578033    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:49.668736 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:49.668677    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86669fc994-n7lns"
Jul 31 23:47:53.011096 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:53.011059    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:53.108043 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:53.108009    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:53.208999 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:53.208968    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:53.546305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:53.546278    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:53.546487 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:53.546321    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:53.550860 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:53.550839    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:53.702726 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:53.702696    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:47:53.746070 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:53.746037    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cf8958db9-phbrr"]
Jul 31 23:47:53.848642 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:53.848574    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:53.905116 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:53.905077    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:53.966842 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:53.966815    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:47:59.068694 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:59.068661    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:47:59.069119 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:47:59.068717    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:48:08.011455 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:08.011414    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:08.069941 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:08.069905    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:16.784214 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:16.784180    2546 generic.go:356] "Generic (PLEG): container finished" podID="4e582be6-7c99-42ca-ad13-a97315764e09" containerID="5dc2c0c29a1462e8c5f72a0c85c6e79a497c1daa646d8a2e97bd1e4e7254d71b" exitCode=0
Jul 31 23:48:16.784592 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:16.784249    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s" event={"ID":"4e582be6-7c99-42ca-ad13-a97315764e09","Type":"ContainerDied","Data":"5dc2c0c29a1462e8c5f72a0c85c6e79a497c1daa646d8a2e97bd1e4e7254d71b"}
Jul 31 23:48:16.784592 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:16.784560    2546 scope.go:117] "RemoveContainer" containerID="5dc2c0c29a1462e8c5f72a0c85c6e79a497c1daa646d8a2e97bd1e4e7254d71b"
Jul 31 23:48:17.789738 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:17.789708    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-sbj9s" event={"ID":"4e582be6-7c99-42ca-ad13-a97315764e09","Type":"ContainerStarted","Data":"cef30b06022f6ec839ef32aaec3ee698ba10e70aeeef70d5f1e980233ec1ab28"}
Jul 31 23:48:18.771208 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:18.771164    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7cf8958db9-phbrr" podUID="404cb752-de98-4920-bbf1-59cc2dd03c05" containerName="console" containerID="cri-o://319f17ae83eeeaa565cf4eec3dbf6538db17dc2910f89b162eba36e0e35a73b8" gracePeriod=15
Jul 31 23:48:19.045282 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.045215    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:48:19.074901 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.074879    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:48:19.078584 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.078565    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6dc885d577-728pn"
Jul 31 23:48:19.138431 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.138410    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/404cb752-de98-4920-bbf1-59cc2dd03c05-console-oauth-config\") pod \"404cb752-de98-4920-bbf1-59cc2dd03c05\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") "
Jul 31 23:48:19.138548 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.138452    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thh7h\" (UniqueName: \"kubernetes.io/projected/404cb752-de98-4920-bbf1-59cc2dd03c05-kube-api-access-thh7h\") pod \"404cb752-de98-4920-bbf1-59cc2dd03c05\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") "
Jul 31 23:48:19.138548 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.138482    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/404cb752-de98-4920-bbf1-59cc2dd03c05-console-serving-cert\") pod \"404cb752-de98-4920-bbf1-59cc2dd03c05\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") "
Jul 31 23:48:19.138548 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.138507    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-oauth-serving-cert\") pod \"404cb752-de98-4920-bbf1-59cc2dd03c05\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") "
Jul 31 23:48:19.138548 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.138531    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-service-ca\") pod \"404cb752-de98-4920-bbf1-59cc2dd03c05\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") "
Jul 31 23:48:19.138761 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.138550    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-trusted-ca-bundle\") pod \"404cb752-de98-4920-bbf1-59cc2dd03c05\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") "
Jul 31 23:48:19.138761 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.138582    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-console-config\") pod \"404cb752-de98-4920-bbf1-59cc2dd03c05\" (UID: \"404cb752-de98-4920-bbf1-59cc2dd03c05\") "
Jul 31 23:48:19.138923 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.138899    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "404cb752-de98-4920-bbf1-59cc2dd03c05" (UID: "404cb752-de98-4920-bbf1-59cc2dd03c05"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:19.139115 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.138997    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-service-ca" (OuterVolumeSpecName: "service-ca") pod "404cb752-de98-4920-bbf1-59cc2dd03c05" (UID: "404cb752-de98-4920-bbf1-59cc2dd03c05"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:19.139208 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.139149    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-console-config" (OuterVolumeSpecName: "console-config") pod "404cb752-de98-4920-bbf1-59cc2dd03c05" (UID: "404cb752-de98-4920-bbf1-59cc2dd03c05"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:19.139394 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.139367    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "404cb752-de98-4920-bbf1-59cc2dd03c05" (UID: "404cb752-de98-4920-bbf1-59cc2dd03c05"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:19.140802 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.140772    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/404cb752-de98-4920-bbf1-59cc2dd03c05-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "404cb752-de98-4920-bbf1-59cc2dd03c05" (UID: "404cb752-de98-4920-bbf1-59cc2dd03c05"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:19.140802 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.140795    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/404cb752-de98-4920-bbf1-59cc2dd03c05-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "404cb752-de98-4920-bbf1-59cc2dd03c05" (UID: "404cb752-de98-4920-bbf1-59cc2dd03c05"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:19.141198 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.141165    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404cb752-de98-4920-bbf1-59cc2dd03c05-kube-api-access-thh7h" (OuterVolumeSpecName: "kube-api-access-thh7h") pod "404cb752-de98-4920-bbf1-59cc2dd03c05" (UID: "404cb752-de98-4920-bbf1-59cc2dd03c05"). InnerVolumeSpecName "kube-api-access-thh7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:48:19.239742 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.239720    2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/404cb752-de98-4920-bbf1-59cc2dd03c05-console-oauth-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:19.239742 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.239741    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thh7h\" (UniqueName: \"kubernetes.io/projected/404cb752-de98-4920-bbf1-59cc2dd03c05-kube-api-access-thh7h\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:19.239869 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.239751    2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/404cb752-de98-4920-bbf1-59cc2dd03c05-console-serving-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:19.239869 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.239760    2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-oauth-serving-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:19.239869 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.239768    2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-service-ca\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:19.239869 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.239776    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-trusted-ca-bundle\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:19.239869 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.239784    2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/404cb752-de98-4920-bbf1-59cc2dd03c05-console-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:19.799235 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.799201    2546 generic.go:356] "Generic (PLEG): container finished" podID="404cb752-de98-4920-bbf1-59cc2dd03c05" containerID="319f17ae83eeeaa565cf4eec3dbf6538db17dc2910f89b162eba36e0e35a73b8" exitCode=0
Jul 31 23:48:19.799428 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.799268    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf8958db9-phbrr"
Jul 31 23:48:19.799428 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.799283    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf8958db9-phbrr" event={"ID":"404cb752-de98-4920-bbf1-59cc2dd03c05","Type":"ContainerDied","Data":"319f17ae83eeeaa565cf4eec3dbf6538db17dc2910f89b162eba36e0e35a73b8"}
Jul 31 23:48:19.799428 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.799334    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf8958db9-phbrr" event={"ID":"404cb752-de98-4920-bbf1-59cc2dd03c05","Type":"ContainerDied","Data":"320fa8faa1ada038540e853d5a9624c18e62a4fc6f6bb56c81c0519690990c54"}
Jul 31 23:48:19.799428 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.799362    2546 scope.go:117] "RemoveContainer" containerID="319f17ae83eeeaa565cf4eec3dbf6538db17dc2910f89b162eba36e0e35a73b8"
Jul 31 23:48:19.808312 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.808009    2546 scope.go:117] "RemoveContainer" containerID="319f17ae83eeeaa565cf4eec3dbf6538db17dc2910f89b162eba36e0e35a73b8"
Jul 31 23:48:19.808500 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:48:19.808450    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319f17ae83eeeaa565cf4eec3dbf6538db17dc2910f89b162eba36e0e35a73b8\": container with ID starting with 319f17ae83eeeaa565cf4eec3dbf6538db17dc2910f89b162eba36e0e35a73b8 not found: ID does not exist" containerID="319f17ae83eeeaa565cf4eec3dbf6538db17dc2910f89b162eba36e0e35a73b8"
Jul 31 23:48:19.808500 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.808491    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319f17ae83eeeaa565cf4eec3dbf6538db17dc2910f89b162eba36e0e35a73b8"} err="failed to get container status \"319f17ae83eeeaa565cf4eec3dbf6538db17dc2910f89b162eba36e0e35a73b8\": rpc error: code = NotFound desc = could not find container \"319f17ae83eeeaa565cf4eec3dbf6538db17dc2910f89b162eba36e0e35a73b8\": container with ID starting with 319f17ae83eeeaa565cf4eec3dbf6538db17dc2910f89b162eba36e0e35a73b8 not found: ID does not exist"
Jul 31 23:48:19.826170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.826151    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cf8958db9-phbrr"]
Jul 31 23:48:19.836423 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:19.836402    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cf8958db9-phbrr"]
Jul 31 23:48:20.074528 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:20.074460    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="404cb752-de98-4920-bbf1-59cc2dd03c05" path="/var/lib/kubelet/pods/404cb752-de98-4920-bbf1-59cc2dd03c05/volumes"
Jul 31 23:48:33.955446 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:33.955412    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:50.407778 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:50.407743    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:50.469811 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:50.469783    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:48:54.664170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.664136    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:48:54.664639 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.664502    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="alertmanager" containerID="cri-o://5d5331dd13a9bdb70bf698fcd5a546cb776bea29b27853ebb96e5d2581cfa07f" gracePeriod=120
Jul 31 23:48:54.664639 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.664541    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="kube-rbac-proxy-metric" containerID="cri-o://13f7fe4aad4908450b840e53b5314ea7d28bc43a784e4571c7645bd87b33cf46" gracePeriod=120
Jul 31 23:48:54.664639 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.664557    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="kube-rbac-proxy-web" containerID="cri-o://6944bac4cf2013b79c14f315c783dcfb4a07bc2881258ee50f69da9ba911882b" gracePeriod=120
Jul 31 23:48:54.664639 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.664583    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="config-reloader" containerID="cri-o://8bc5a5884d7cf7d3f0aa81b8e2a6e27c5d370e5d1ee4c4553d0a4e8a2cf3a34a" gracePeriod=120
Jul 31 23:48:54.664837 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.664659    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="prom-label-proxy" containerID="cri-o://c72d84957629906ed1b2770083009292292e0ff7ec20456780c52942957855d1" gracePeriod=120
Jul 31 23:48:54.664837 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.664596    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="kube-rbac-proxy" containerID="cri-o://9b7a7b360b42b4deb975ff78cfb33fa86b80704d0420cb3ebb083769005177f0" gracePeriod=120
Jul 31 23:48:54.941263 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.941176    2546 generic.go:356] "Generic (PLEG): container finished" podID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerID="c72d84957629906ed1b2770083009292292e0ff7ec20456780c52942957855d1" exitCode=0
Jul 31 23:48:54.941263 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.941207    2546 generic.go:356] "Generic (PLEG): container finished" podID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerID="13f7fe4aad4908450b840e53b5314ea7d28bc43a784e4571c7645bd87b33cf46" exitCode=0
Jul 31 23:48:54.941263 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.941214    2546 generic.go:356] "Generic (PLEG): container finished" podID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerID="9b7a7b360b42b4deb975ff78cfb33fa86b80704d0420cb3ebb083769005177f0" exitCode=0
Jul 31 23:48:54.941263 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.941219    2546 generic.go:356] "Generic (PLEG): container finished" podID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerID="8bc5a5884d7cf7d3f0aa81b8e2a6e27c5d370e5d1ee4c4553d0a4e8a2cf3a34a" exitCode=0
Jul 31 23:48:54.941263 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.941224    2546 generic.go:356] "Generic (PLEG): container finished" podID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerID="5d5331dd13a9bdb70bf698fcd5a546cb776bea29b27853ebb96e5d2581cfa07f" exitCode=0
Jul 31 23:48:54.941501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.941246    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3","Type":"ContainerDied","Data":"c72d84957629906ed1b2770083009292292e0ff7ec20456780c52942957855d1"}
Jul 31 23:48:54.941501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.941324    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3","Type":"ContainerDied","Data":"13f7fe4aad4908450b840e53b5314ea7d28bc43a784e4571c7645bd87b33cf46"}
Jul 31 23:48:54.941501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.941352    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3","Type":"ContainerDied","Data":"9b7a7b360b42b4deb975ff78cfb33fa86b80704d0420cb3ebb083769005177f0"}
Jul 31 23:48:54.941501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.941377    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3","Type":"ContainerDied","Data":"8bc5a5884d7cf7d3f0aa81b8e2a6e27c5d370e5d1ee4c4553d0a4e8a2cf3a34a"}
Jul 31 23:48:54.941501 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:54.941401    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3","Type":"ContainerDied","Data":"5d5331dd13a9bdb70bf698fcd5a546cb776bea29b27853ebb96e5d2581cfa07f"}
Jul 31 23:48:55.902536 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.902515    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:55.950154 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.950121    2546 generic.go:356] "Generic (PLEG): container finished" podID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerID="6944bac4cf2013b79c14f315c783dcfb4a07bc2881258ee50f69da9ba911882b" exitCode=0
Jul 31 23:48:55.950302 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.950165    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3","Type":"ContainerDied","Data":"6944bac4cf2013b79c14f315c783dcfb4a07bc2881258ee50f69da9ba911882b"}
Jul 31 23:48:55.950302 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.950203    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:55.950302 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.950241    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3","Type":"ContainerDied","Data":"dc29fa637b7704d933f96fb3ba3b8976a40bb565af7cc234155128b19136f061"}
Jul 31 23:48:55.950302 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.950282    2546 scope.go:117] "RemoveContainer" containerID="c72d84957629906ed1b2770083009292292e0ff7ec20456780c52942957855d1"
Jul 31 23:48:55.959073 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.959047    2546 scope.go:117] "RemoveContainer" containerID="13f7fe4aad4908450b840e53b5314ea7d28bc43a784e4571c7645bd87b33cf46"
Jul 31 23:48:55.965756 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.965739    2546 scope.go:117] "RemoveContainer" containerID="9b7a7b360b42b4deb975ff78cfb33fa86b80704d0420cb3ebb083769005177f0"
Jul 31 23:48:55.972037 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.972021    2546 scope.go:117] "RemoveContainer" containerID="6944bac4cf2013b79c14f315c783dcfb4a07bc2881258ee50f69da9ba911882b"
Jul 31 23:48:55.978164 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.978147    2546 scope.go:117] "RemoveContainer" containerID="8bc5a5884d7cf7d3f0aa81b8e2a6e27c5d370e5d1ee4c4553d0a4e8a2cf3a34a"
Jul 31 23:48:55.984921 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.984893    2546 scope.go:117] "RemoveContainer" containerID="5d5331dd13a9bdb70bf698fcd5a546cb776bea29b27853ebb96e5d2581cfa07f"
Jul 31 23:48:55.991554 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.991538    2546 scope.go:117] "RemoveContainer" containerID="de5ca77dee7bf355a5e1ee6a94cc514bf2301f06f2d532307ca2e94745673efa"
Jul 31 23:48:55.997941 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.997925    2546 scope.go:117] "RemoveContainer" containerID="c72d84957629906ed1b2770083009292292e0ff7ec20456780c52942957855d1"
Jul 31 23:48:55.998197 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:48:55.998179    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72d84957629906ed1b2770083009292292e0ff7ec20456780c52942957855d1\": container with ID starting with c72d84957629906ed1b2770083009292292e0ff7ec20456780c52942957855d1 not found: ID does not exist" containerID="c72d84957629906ed1b2770083009292292e0ff7ec20456780c52942957855d1"
Jul 31 23:48:55.998260 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.998206    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72d84957629906ed1b2770083009292292e0ff7ec20456780c52942957855d1"} err="failed to get container status \"c72d84957629906ed1b2770083009292292e0ff7ec20456780c52942957855d1\": rpc error: code = NotFound desc = could not find container \"c72d84957629906ed1b2770083009292292e0ff7ec20456780c52942957855d1\": container with ID starting with c72d84957629906ed1b2770083009292292e0ff7ec20456780c52942957855d1 not found: ID does not exist"
Jul 31 23:48:55.998260 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.998224    2546 scope.go:117] "RemoveContainer" containerID="13f7fe4aad4908450b840e53b5314ea7d28bc43a784e4571c7645bd87b33cf46"
Jul 31 23:48:55.998511 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:48:55.998493    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f7fe4aad4908450b840e53b5314ea7d28bc43a784e4571c7645bd87b33cf46\": container with ID starting with 13f7fe4aad4908450b840e53b5314ea7d28bc43a784e4571c7645bd87b33cf46 not found: ID does not exist" containerID="13f7fe4aad4908450b840e53b5314ea7d28bc43a784e4571c7645bd87b33cf46"
Jul 31 23:48:55.998564 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.998515    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f7fe4aad4908450b840e53b5314ea7d28bc43a784e4571c7645bd87b33cf46"} err="failed to get container status \"13f7fe4aad4908450b840e53b5314ea7d28bc43a784e4571c7645bd87b33cf46\": rpc error: code = NotFound desc = could not find container \"13f7fe4aad4908450b840e53b5314ea7d28bc43a784e4571c7645bd87b33cf46\": container with ID starting with 13f7fe4aad4908450b840e53b5314ea7d28bc43a784e4571c7645bd87b33cf46 not found: ID does not exist"
Jul 31 23:48:55.998564 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.998530    2546 scope.go:117] "RemoveContainer" containerID="9b7a7b360b42b4deb975ff78cfb33fa86b80704d0420cb3ebb083769005177f0"
Jul 31 23:48:55.998739 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:48:55.998723    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7a7b360b42b4deb975ff78cfb33fa86b80704d0420cb3ebb083769005177f0\": container with ID starting with 9b7a7b360b42b4deb975ff78cfb33fa86b80704d0420cb3ebb083769005177f0 not found: ID does not exist" containerID="9b7a7b360b42b4deb975ff78cfb33fa86b80704d0420cb3ebb083769005177f0"
Jul 31 23:48:55.998779 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.998743    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7a7b360b42b4deb975ff78cfb33fa86b80704d0420cb3ebb083769005177f0"} err="failed to get container status \"9b7a7b360b42b4deb975ff78cfb33fa86b80704d0420cb3ebb083769005177f0\": rpc error: code = NotFound desc = could not find container \"9b7a7b360b42b4deb975ff78cfb33fa86b80704d0420cb3ebb083769005177f0\": container with ID starting with 9b7a7b360b42b4deb975ff78cfb33fa86b80704d0420cb3ebb083769005177f0 not found: ID does not exist"
Jul 31 23:48:55.998779 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.998756    2546 scope.go:117] "RemoveContainer" containerID="6944bac4cf2013b79c14f315c783dcfb4a07bc2881258ee50f69da9ba911882b"
Jul 31 23:48:55.998981 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:48:55.998961    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6944bac4cf2013b79c14f315c783dcfb4a07bc2881258ee50f69da9ba911882b\": container with ID starting with 6944bac4cf2013b79c14f315c783dcfb4a07bc2881258ee50f69da9ba911882b not found: ID does not exist" containerID="6944bac4cf2013b79c14f315c783dcfb4a07bc2881258ee50f69da9ba911882b"
Jul 31 23:48:55.999043 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.998988    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6944bac4cf2013b79c14f315c783dcfb4a07bc2881258ee50f69da9ba911882b"} err="failed to get container status \"6944bac4cf2013b79c14f315c783dcfb4a07bc2881258ee50f69da9ba911882b\": rpc error: code = NotFound desc = could not find container \"6944bac4cf2013b79c14f315c783dcfb4a07bc2881258ee50f69da9ba911882b\": container with ID starting with 6944bac4cf2013b79c14f315c783dcfb4a07bc2881258ee50f69da9ba911882b not found: ID does not exist"
Jul 31 23:48:55.999043 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.999008    2546 scope.go:117] "RemoveContainer" containerID="8bc5a5884d7cf7d3f0aa81b8e2a6e27c5d370e5d1ee4c4553d0a4e8a2cf3a34a"
Jul 31 23:48:55.999218 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:48:55.999203    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc5a5884d7cf7d3f0aa81b8e2a6e27c5d370e5d1ee4c4553d0a4e8a2cf3a34a\": container with ID starting with 8bc5a5884d7cf7d3f0aa81b8e2a6e27c5d370e5d1ee4c4553d0a4e8a2cf3a34a not found: ID does not exist" containerID="8bc5a5884d7cf7d3f0aa81b8e2a6e27c5d370e5d1ee4c4553d0a4e8a2cf3a34a"
Jul 31 23:48:55.999268 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.999221    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc5a5884d7cf7d3f0aa81b8e2a6e27c5d370e5d1ee4c4553d0a4e8a2cf3a34a"} err="failed to get container status \"8bc5a5884d7cf7d3f0aa81b8e2a6e27c5d370e5d1ee4c4553d0a4e8a2cf3a34a\": rpc error: code = NotFound desc = could not find container \"8bc5a5884d7cf7d3f0aa81b8e2a6e27c5d370e5d1ee4c4553d0a4e8a2cf3a34a\": container with ID starting with 8bc5a5884d7cf7d3f0aa81b8e2a6e27c5d370e5d1ee4c4553d0a4e8a2cf3a34a not found: ID does not exist"
Jul 31 23:48:55.999268 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.999232    2546 scope.go:117] "RemoveContainer" containerID="5d5331dd13a9bdb70bf698fcd5a546cb776bea29b27853ebb96e5d2581cfa07f"
Jul 31 23:48:55.999524 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:48:55.999509    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5331dd13a9bdb70bf698fcd5a546cb776bea29b27853ebb96e5d2581cfa07f\": container with ID starting with 5d5331dd13a9bdb70bf698fcd5a546cb776bea29b27853ebb96e5d2581cfa07f not found: ID does not exist" containerID="5d5331dd13a9bdb70bf698fcd5a546cb776bea29b27853ebb96e5d2581cfa07f"
Jul 31 23:48:55.999569 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.999527    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5331dd13a9bdb70bf698fcd5a546cb776bea29b27853ebb96e5d2581cfa07f"} err="failed to get container status \"5d5331dd13a9bdb70bf698fcd5a546cb776bea29b27853ebb96e5d2581cfa07f\": rpc error: code = NotFound desc = could not find container \"5d5331dd13a9bdb70bf698fcd5a546cb776bea29b27853ebb96e5d2581cfa07f\": container with ID starting with 5d5331dd13a9bdb70bf698fcd5a546cb776bea29b27853ebb96e5d2581cfa07f not found: ID does not exist"
Jul 31 23:48:55.999569 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.999541    2546 scope.go:117] "RemoveContainer" containerID="de5ca77dee7bf355a5e1ee6a94cc514bf2301f06f2d532307ca2e94745673efa"
Jul 31 23:48:55.999754 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:48:55.999737    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5ca77dee7bf355a5e1ee6a94cc514bf2301f06f2d532307ca2e94745673efa\": container with ID starting with de5ca77dee7bf355a5e1ee6a94cc514bf2301f06f2d532307ca2e94745673efa not found: ID does not exist" containerID="de5ca77dee7bf355a5e1ee6a94cc514bf2301f06f2d532307ca2e94745673efa"
Jul 31 23:48:55.999795 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:55.999760    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5ca77dee7bf355a5e1ee6a94cc514bf2301f06f2d532307ca2e94745673efa"} err="failed to get container status \"de5ca77dee7bf355a5e1ee6a94cc514bf2301f06f2d532307ca2e94745673efa\": rpc error: code = NotFound desc = could not find container \"de5ca77dee7bf355a5e1ee6a94cc514bf2301f06f2d532307ca2e94745673efa\": container with ID starting with de5ca77dee7bf355a5e1ee6a94cc514bf2301f06f2d532307ca2e94745673efa not found: ID does not exist"
Jul 31 23:48:56.020005 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.019985    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-tls-assets\") pod \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") "
Jul 31 23:48:56.020077 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.020015    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-cluster-tls-config\") pod \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") "
Jul 31 23:48:56.020077 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.020038    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-metrics-client-ca\") pod \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") "
Jul 31 23:48:56.020077 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.020056    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddtsc\" (UniqueName: \"kubernetes.io/projected/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-kube-api-access-ddtsc\") pod \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") "
Jul 31 23:48:56.020228 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.020081    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") "
Jul 31 23:48:56.020228 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.020101    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy\") pod \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") "
Jul 31 23:48:56.020228 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.020137    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-main-tls\") pod \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") "
Jul 31 23:48:56.020228 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.020174    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-config-volume\") pod \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") "
Jul 31 23:48:56.020228 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.020209    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-alertmanager-main-db\") pod \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") "
Jul 31 23:48:56.020545 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.020301    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-web-config\") pod \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") "
Jul 31 23:48:56.020545 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.020344    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-config-out\") pod \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") "
Jul 31 23:48:56.020545 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.020389    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-alertmanager-trusted-ca-bundle\") pod \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") "
Jul 31 23:48:56.020545 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.020417    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\" (UID: \"c69b96d7-838c-4c38-88e8-c4ab90f0d2f3\") "
Jul 31 23:48:56.020545 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.020461    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" (UID: "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:56.020789 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.020671    2546 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-metrics-client-ca\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:56.020946 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.020924    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" (UID: "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:48:56.022140 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.022111    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" (UID: "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:48:56.023969 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.023931    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" (UID: "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:48:56.024059 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.023980    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" (UID: "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:56.024059 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.024033    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-config-volume" (OuterVolumeSpecName: "config-volume") pod "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" (UID: "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:56.024059 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.024038    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" (UID: "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:56.024307 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.024246    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" (UID: "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:56.024479 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.024429    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" (UID: "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:56.024479 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.024440    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-config-out" (OuterVolumeSpecName: "config-out") pod "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" (UID: "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:48:56.024583 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.024519    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-kube-api-access-ddtsc" (OuterVolumeSpecName: "kube-api-access-ddtsc") pod "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" (UID: "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3"). InnerVolumeSpecName "kube-api-access-ddtsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:48:56.027189 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.027162    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" (UID: "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:56.033511 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.033489    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-web-config" (OuterVolumeSpecName: "web-config") pod "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" (UID: "c69b96d7-838c-4c38-88e8-c4ab90f0d2f3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:48:56.121909 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.121885    2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-main-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:56.121909 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.121906    2546 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-config-volume\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:56.122018 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.121915    2546 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-alertmanager-main-db\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:56.122018 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.121923    2546 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-web-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:56.122018 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.121931    2546 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-config-out\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:56.122018 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.121939    2546 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:56.122018 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.121948    2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:56.122018 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.121956    2546 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-tls-assets\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:56.122018 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.121965    2546 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-cluster-tls-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:56.122018 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.121973    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddtsc\" (UniqueName: \"kubernetes.io/projected/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-kube-api-access-ddtsc\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:56.122018 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.121981    2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:56.122018 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.121990    2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:48:56.274214 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.274141    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:48:56.277351 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.277326    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:48:56.301260 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301233    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:48:56.301579 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301562    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="alertmanager"
Jul 31 23:48:56.301646 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301582    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="alertmanager"
Jul 31 23:48:56.301646 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301596    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="kube-rbac-proxy-web"
Jul 31 23:48:56.301646 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301605    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="kube-rbac-proxy-web"
Jul 31 23:48:56.301646 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301616    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="prom-label-proxy"
Jul 31 23:48:56.301646 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301624    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="prom-label-proxy"
Jul 31 23:48:56.301646 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301645    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="kube-rbac-proxy-metric"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301654    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="kube-rbac-proxy-metric"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301669    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="404cb752-de98-4920-bbf1-59cc2dd03c05" containerName="console"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301677    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="404cb752-de98-4920-bbf1-59cc2dd03c05" containerName="console"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301688    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="init-config-reloader"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301697    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="init-config-reloader"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301708    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="config-reloader"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301716    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="config-reloader"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301725    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="kube-rbac-proxy"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301734    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="kube-rbac-proxy"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301812    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="kube-rbac-proxy-metric"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301824    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="prom-label-proxy"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301834    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="config-reloader"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301844    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="kube-rbac-proxy-web"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301854    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="404cb752-de98-4920-bbf1-59cc2dd03c05" containerName="console"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301864    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="alertmanager"
Jul 31 23:48:56.301936 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.301874    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" containerName="kube-rbac-proxy"
Jul 31 23:48:56.305642 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.305625    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.308206 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.308187    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 31 23:48:56.308206 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.308200    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 31 23:48:56.308386 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.308296    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 31 23:48:56.308386 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.308192    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 31 23:48:56.308386 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.308311    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 31 23:48:56.308527 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.308447    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 31 23:48:56.308577 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.308534    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 31 23:48:56.308577 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.308552    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 31 23:48:56.308577 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.308560    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ppj72\""
Jul 31 23:48:56.313603 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.313581    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 31 23:48:56.319568 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.319551    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:48:56.423753 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.423725    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.423917 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.423764    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.423917 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.423787    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/425aeefe-ddeb-46e9-809d-5563e37b13f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.423917 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.423841    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-web-config\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.423917 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.423864    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425aeefe-ddeb-46e9-809d-5563e37b13f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.423917 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.423901    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/425aeefe-ddeb-46e9-809d-5563e37b13f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.424170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.423933    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.424170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.423984    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.424170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.424009    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/425aeefe-ddeb-46e9-809d-5563e37b13f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.424170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.424035    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.424170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.424063    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/425aeefe-ddeb-46e9-809d-5563e37b13f0-config-out\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.424170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.424121    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcqxg\" (UniqueName: \"kubernetes.io/projected/425aeefe-ddeb-46e9-809d-5563e37b13f0-kube-api-access-tcqxg\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.424170 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.424155    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.524667 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.524605    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.524667 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.524638    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.524667 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.524655    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/425aeefe-ddeb-46e9-809d-5563e37b13f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.524859 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.524775    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.524859 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.524811    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/425aeefe-ddeb-46e9-809d-5563e37b13f0-config-out\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.524859 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.524841    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcqxg\" (UniqueName: \"kubernetes.io/projected/425aeefe-ddeb-46e9-809d-5563e37b13f0-kube-api-access-tcqxg\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.525011 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.524879    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.525011 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.524918    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.525011 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.524949    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.525011 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.524975    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/425aeefe-ddeb-46e9-809d-5563e37b13f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.525225 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.525013    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-web-config\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.525225 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.525041    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425aeefe-ddeb-46e9-809d-5563e37b13f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.525225 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.525088    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/425aeefe-ddeb-46e9-809d-5563e37b13f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.525848 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.525823    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/425aeefe-ddeb-46e9-809d-5563e37b13f0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.525952 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.525891    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/425aeefe-ddeb-46e9-809d-5563e37b13f0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.527946 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.527557    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/425aeefe-ddeb-46e9-809d-5563e37b13f0-config-out\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.527946 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.527635    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.527946 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.527643    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.527946 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.527653    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-config-volume\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.527946 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.527785    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/425aeefe-ddeb-46e9-809d-5563e37b13f0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.527946 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.527942    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.528336 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.528218    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.528336 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.528290    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-web-config\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.528421 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.528343    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425aeefe-ddeb-46e9-809d-5563e37b13f0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.529464 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.529447    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/425aeefe-ddeb-46e9-809d-5563e37b13f0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.533360 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.533342    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcqxg\" (UniqueName: \"kubernetes.io/projected/425aeefe-ddeb-46e9-809d-5563e37b13f0-kube-api-access-tcqxg\") pod \"alertmanager-main-0\" (UID: \"425aeefe-ddeb-46e9-809d-5563e37b13f0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.615392 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.615371    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 23:48:56.748934 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.748909    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 23:48:56.751094 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:48:56.751067    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425aeefe_ddeb_46e9_809d_5563e37b13f0.slice/crio-251ce5c9f310319b596ea4b8463092fe02fe8fc9d2b7f76cf2f75ac0f3615a73 WatchSource:0}: Error finding container 251ce5c9f310319b596ea4b8463092fe02fe8fc9d2b7f76cf2f75ac0f3615a73: Status 404 returned error can't find the container with id 251ce5c9f310319b596ea4b8463092fe02fe8fc9d2b7f76cf2f75ac0f3615a73
Jul 31 23:48:56.955351 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.955320    2546 generic.go:356] "Generic (PLEG): container finished" podID="425aeefe-ddeb-46e9-809d-5563e37b13f0" containerID="d6556e821705c590e803236da5f9c2563c358f5daabbfc14cb24d4250e807ce3" exitCode=0
Jul 31 23:48:56.955735 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.955406    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"425aeefe-ddeb-46e9-809d-5563e37b13f0","Type":"ContainerDied","Data":"d6556e821705c590e803236da5f9c2563c358f5daabbfc14cb24d4250e807ce3"}
Jul 31 23:48:56.955735 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:56.955475    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"425aeefe-ddeb-46e9-809d-5563e37b13f0","Type":"ContainerStarted","Data":"251ce5c9f310319b596ea4b8463092fe02fe8fc9d2b7f76cf2f75ac0f3615a73"}
Jul 31 23:48:57.964088 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:57.964050    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"425aeefe-ddeb-46e9-809d-5563e37b13f0","Type":"ContainerStarted","Data":"0ac87301f9d4f1c30074f15b66da115150fdd2413bd12a7678c80c69305df77f"}
Jul 31 23:48:57.964471 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:57.964129    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"425aeefe-ddeb-46e9-809d-5563e37b13f0","Type":"ContainerStarted","Data":"3fcf4e525f148a200785f0b58316c1535bfc928120e159b30062a41c62e5bfdb"}
Jul 31 23:48:57.964471 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:57.964170    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"425aeefe-ddeb-46e9-809d-5563e37b13f0","Type":"ContainerStarted","Data":"2266b4a663f4e6343a8abab20c2af1dd929a47817e6b0707cc1aad780bd23b6e"}
Jul 31 23:48:57.964471 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:57.964206    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"425aeefe-ddeb-46e9-809d-5563e37b13f0","Type":"ContainerStarted","Data":"843203e7961cbfeddb364fc0bfef69c8d17b4552f20f53af0548871f44e6da1b"}
Jul 31 23:48:57.964471 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:57.964241    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"425aeefe-ddeb-46e9-809d-5563e37b13f0","Type":"ContainerStarted","Data":"e6a2f000735a8c0e57e409ae2f01aefe5343aa0f75695c2f5d2a675f5e34160d"}
Jul 31 23:48:57.964471 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:57.964299    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"425aeefe-ddeb-46e9-809d-5563e37b13f0","Type":"ContainerStarted","Data":"563e736c284dd837bf11718e12fb8ab9015044538e8dbb3b8417aa449a369b10"}
Jul 31 23:48:57.991714 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:57.991671    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.971760379 podStartE2EDuration="1.991648982s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.888603ms" isStatefulPod=false podCreationTimestamp="2026-07-31 23:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:48:57.99050952 +0000 UTC m=+168.449972337" watchObservedRunningTime="2026-07-31 23:48:57.991648982 +0000 UTC m=+168.451111795"
Jul 31 23:48:58.076096 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:48:58.076069    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c69b96d7-838c-4c38-88e8-c4ab90f0d2f3" path="/var/lib/kubelet/pods/c69b96d7-838c-4c38-88e8-c4ab90f0d2f3/volumes"
Jul 31 23:49:02.386296 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:02.386191    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:02.457524 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:02.457491    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:08.019572 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:08.019534    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:08.119933 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:08.119894    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:08.173419 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:08.173391    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:10.852167 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:10.852132    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f8f79dcf4-qxj2w"]
Jul 31 23:49:10.855093 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:10.855071    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:10.864126 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:10.864104    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f8f79dcf4-qxj2w"]
Jul 31 23:49:10.908212 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:10.908188    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:10.935340 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:10.935318    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-trusted-ca-bundle\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:10.935413 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:10.935351    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-oauth-config\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:10.935413 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:10.935368    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-service-ca\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:10.935485 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:10.935453    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-config\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:10.935485 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:10.935475    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-oauth-serving-cert\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:10.935551 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:10.935512    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qnbn\" (UniqueName: \"kubernetes.io/projected/50cbd0c3-2433-4666-8e6c-d33e505ec692-kube-api-access-6qnbn\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:10.935596 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:10.935583    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-serving-cert\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:10.963241 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:10.963201    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:11.036535 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.036510    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-config\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:11.036642 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.036537    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-oauth-serving-cert\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:11.036642 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.036565    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qnbn\" (UniqueName: \"kubernetes.io/projected/50cbd0c3-2433-4666-8e6c-d33e505ec692-kube-api-access-6qnbn\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:11.036642 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.036597    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-serving-cert\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:11.036642 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.036614    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-trusted-ca-bundle\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:11.036642 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.036633    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-oauth-config\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:11.036894 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.036649    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-service-ca\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:11.037289 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.037269    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-oauth-serving-cert\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:11.037396 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.037365    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-config\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:11.037725 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.037701    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-service-ca\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:11.037816 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.037713    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-trusted-ca-bundle\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:11.039236 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.039215    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-oauth-config\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:11.039425 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.039410    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-serving-cert\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:11.046537 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.046515    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qnbn\" (UniqueName: \"kubernetes.io/projected/50cbd0c3-2433-4666-8e6c-d33e505ec692-kube-api-access-6qnbn\") pod \"console-6f8f79dcf4-qxj2w\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") " pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:11.068592 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.068573    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:11.164281 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.164241    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:11.170237 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.170214    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:11.286034 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.285829    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f8f79dcf4-qxj2w"]
Jul 31 23:49:11.288773 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:49:11.288744    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cbd0c3_2433_4666_8e6c_d33e505ec692.slice/crio-787f35b3a3be73d28730c0ee5e40c502db0fe56d99693375ab3aeb14590700d7 WatchSource:0}: Error finding container 787f35b3a3be73d28730c0ee5e40c502db0fe56d99693375ab3aeb14590700d7: Status 404 returned error can't find the container with id 787f35b3a3be73d28730c0ee5e40c502db0fe56d99693375ab3aeb14590700d7
Jul 31 23:49:11.870410 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:11.870358    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:12.015957 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:12.015922    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8f79dcf4-qxj2w" event={"ID":"50cbd0c3-2433-4666-8e6c-d33e505ec692","Type":"ContainerStarted","Data":"736fd1c93eafeee40136ee516750d8eb145888924cfba84ced955d4acea28606"}
Jul 31 23:49:12.016089 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:12.015970    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8f79dcf4-qxj2w" event={"ID":"50cbd0c3-2433-4666-8e6c-d33e505ec692","Type":"ContainerStarted","Data":"787f35b3a3be73d28730c0ee5e40c502db0fe56d99693375ab3aeb14590700d7"}
Jul 31 23:49:12.033593 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:12.033464    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f8f79dcf4-qxj2w" podStartSLOduration=2.033446674 podStartE2EDuration="2.033446674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:49:12.032240736 +0000 UTC m=+182.491703552" watchObservedRunningTime="2026-07-31 23:49:12.033446674 +0000 UTC m=+182.492909487"
Jul 31 23:49:21.164754 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:21.164709    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:21.165238 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:21.164780    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:21.169523 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:21.169503    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:22.055528 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:22.055499    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:49:22.098945 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:22.098916    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bdc56cf6b-598vg"]
Jul 31 23:49:22.200826 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:22.200792    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:22.256715 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:22.256679    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:22.313142 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:22.313058    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:49:47.120112 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.120075    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bdc56cf6b-598vg" podUID="ffd8234c-196e-437d-922a-8452c780c3aa" containerName="console" containerID="cri-o://54de3cd581547cd11d920a16c5f3adcb7aa7174adf731067e7d3a5c061d3bd43" gracePeriod=15
Jul 31 23:49:47.355389 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.355367    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:49:47.421155 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.421096    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-trusted-ca-bundle\") pod \"ffd8234c-196e-437d-922a-8452c780c3aa\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") "
Jul 31 23:49:47.421155 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.421135    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-service-ca\") pod \"ffd8234c-196e-437d-922a-8452c780c3aa\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") "
Jul 31 23:49:47.421316 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.421170    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-oauth-serving-cert\") pod \"ffd8234c-196e-437d-922a-8452c780c3aa\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") "
Jul 31 23:49:47.421353 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.421322    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffd8234c-196e-437d-922a-8452c780c3aa-console-serving-cert\") pod \"ffd8234c-196e-437d-922a-8452c780c3aa\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") "
Jul 31 23:49:47.421434 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.421404    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffd8234c-196e-437d-922a-8452c780c3aa-console-oauth-config\") pod \"ffd8234c-196e-437d-922a-8452c780c3aa\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") "
Jul 31 23:49:47.421486 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.421433    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-console-config\") pod \"ffd8234c-196e-437d-922a-8452c780c3aa\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") "
Jul 31 23:49:47.421486 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.421459    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rf26\" (UniqueName: \"kubernetes.io/projected/ffd8234c-196e-437d-922a-8452c780c3aa-kube-api-access-7rf26\") pod \"ffd8234c-196e-437d-922a-8452c780c3aa\" (UID: \"ffd8234c-196e-437d-922a-8452c780c3aa\") "
Jul 31 23:49:47.421588 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.421508    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ffd8234c-196e-437d-922a-8452c780c3aa" (UID: "ffd8234c-196e-437d-922a-8452c780c3aa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:47.421588 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.421532    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ffd8234c-196e-437d-922a-8452c780c3aa" (UID: "ffd8234c-196e-437d-922a-8452c780c3aa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:47.421588 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.421541    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-service-ca" (OuterVolumeSpecName: "service-ca") pod "ffd8234c-196e-437d-922a-8452c780c3aa" (UID: "ffd8234c-196e-437d-922a-8452c780c3aa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:47.421847 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.421819    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-console-config" (OuterVolumeSpecName: "console-config") pod "ffd8234c-196e-437d-922a-8452c780c3aa" (UID: "ffd8234c-196e-437d-922a-8452c780c3aa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:49:47.421959 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.421881    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-trusted-ca-bundle\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:49:47.421959 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.421903    2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-service-ca\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:49:47.421959 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.421919    2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-oauth-serving-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:49:47.423365 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.423338    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd8234c-196e-437d-922a-8452c780c3aa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ffd8234c-196e-437d-922a-8452c780c3aa" (UID: "ffd8234c-196e-437d-922a-8452c780c3aa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:49:47.423505 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.423486    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd8234c-196e-437d-922a-8452c780c3aa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ffd8234c-196e-437d-922a-8452c780c3aa" (UID: "ffd8234c-196e-437d-922a-8452c780c3aa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:49:47.423635 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.423616    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd8234c-196e-437d-922a-8452c780c3aa-kube-api-access-7rf26" (OuterVolumeSpecName: "kube-api-access-7rf26") pod "ffd8234c-196e-437d-922a-8452c780c3aa" (UID: "ffd8234c-196e-437d-922a-8452c780c3aa"). InnerVolumeSpecName "kube-api-access-7rf26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:49:47.522592 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.522563    2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffd8234c-196e-437d-922a-8452c780c3aa-console-serving-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:49:47.522592 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.522581    2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffd8234c-196e-437d-922a-8452c780c3aa-console-oauth-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:49:47.522592 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.522591    2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffd8234c-196e-437d-922a-8452c780c3aa-console-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:49:47.522768 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:47.522603    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rf26\" (UniqueName: \"kubernetes.io/projected/ffd8234c-196e-437d-922a-8452c780c3aa-kube-api-access-7rf26\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:49:48.148814 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:48.148781    2546 generic.go:356] "Generic (PLEG): container finished" podID="ffd8234c-196e-437d-922a-8452c780c3aa" containerID="54de3cd581547cd11d920a16c5f3adcb7aa7174adf731067e7d3a5c061d3bd43" exitCode=0
Jul 31 23:49:48.149164 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:48.148840    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bdc56cf6b-598vg"
Jul 31 23:49:48.149164 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:48.148860    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bdc56cf6b-598vg" event={"ID":"ffd8234c-196e-437d-922a-8452c780c3aa","Type":"ContainerDied","Data":"54de3cd581547cd11d920a16c5f3adcb7aa7174adf731067e7d3a5c061d3bd43"}
Jul 31 23:49:48.149164 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:48.148906    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bdc56cf6b-598vg" event={"ID":"ffd8234c-196e-437d-922a-8452c780c3aa","Type":"ContainerDied","Data":"c00b4f472ce7fd427bc5e08a92f1c21b0d81ca71ead9d5255c9381e8a154d1df"}
Jul 31 23:49:48.149164 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:48.148933    2546 scope.go:117] "RemoveContainer" containerID="54de3cd581547cd11d920a16c5f3adcb7aa7174adf731067e7d3a5c061d3bd43"
Jul 31 23:49:48.156697 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:48.156681    2546 scope.go:117] "RemoveContainer" containerID="54de3cd581547cd11d920a16c5f3adcb7aa7174adf731067e7d3a5c061d3bd43"
Jul 31 23:49:48.156973 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:49:48.156948    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54de3cd581547cd11d920a16c5f3adcb7aa7174adf731067e7d3a5c061d3bd43\": container with ID starting with 54de3cd581547cd11d920a16c5f3adcb7aa7174adf731067e7d3a5c061d3bd43 not found: ID does not exist" containerID="54de3cd581547cd11d920a16c5f3adcb7aa7174adf731067e7d3a5c061d3bd43"
Jul 31 23:49:48.157018 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:48.156980    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54de3cd581547cd11d920a16c5f3adcb7aa7174adf731067e7d3a5c061d3bd43"} err="failed to get container status \"54de3cd581547cd11d920a16c5f3adcb7aa7174adf731067e7d3a5c061d3bd43\": rpc error: code = NotFound desc = could not find container \"54de3cd581547cd11d920a16c5f3adcb7aa7174adf731067e7d3a5c061d3bd43\": container with ID starting with 54de3cd581547cd11d920a16c5f3adcb7aa7174adf731067e7d3a5c061d3bd43 not found: ID does not exist"
Jul 31 23:49:48.164542 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:48.164523    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bdc56cf6b-598vg"]
Jul 31 23:49:48.168033 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:48.168009    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bdc56cf6b-598vg"]
Jul 31 23:49:50.075677 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:50.075646    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffd8234c-196e-437d-922a-8452c780c3aa" path="/var/lib/kubelet/pods/ffd8234c-196e-437d-922a-8452c780c3aa/volumes"
Jul 31 23:49:54.141823 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:49:54.141776    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:01.968388 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:01.968349    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:02.027125 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:02.027091    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:17.190630 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:17.190589    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:22.088481 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:22.088441    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:22.139166 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:22.139137    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:22.217028 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:22.216982    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:44.104139 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:44.104107    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:44.198100 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:44.198065    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:48.908518 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:48.908480    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:49.200315 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.200217    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:49.301244 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.301215    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:49.443082 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.443054    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:49.466326 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.466267    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"]
Jul 31 23:50:49.466557 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.466546    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd8234c-196e-437d-922a-8452c780c3aa" containerName="console"
Jul 31 23:50:49.466598 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.466559    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd8234c-196e-437d-922a-8452c780c3aa" containerName="console"
Jul 31 23:50:49.466631 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.466620    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffd8234c-196e-437d-922a-8452c780c3aa" containerName="console"
Jul 31 23:50:49.469378 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.469361    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"
Jul 31 23:50:49.473262 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.473227    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 31 23:50:49.473262 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.473234    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 31 23:50:49.474607 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.474544    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 31 23:50:49.474715 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.474660    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-sbch5\""
Jul 31 23:50:49.474771 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.474714    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 31 23:50:49.474935 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.474892    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 31 23:50:49.480915 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.480895    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"]
Jul 31 23:50:49.482439 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.482418    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4qcm\" (UniqueName: \"kubernetes.io/projected/2df193ce-4a89-4a28-9442-eb9639c22677-kube-api-access-n4qcm\") pod \"keda-metrics-apiserver-55557df8c-8kq6r\" (UID: \"2df193ce-4a89-4a28-9442-eb9639c22677\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"
Jul 31 23:50:49.482536 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.482451    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2df193ce-4a89-4a28-9442-eb9639c22677-certificates\") pod \"keda-metrics-apiserver-55557df8c-8kq6r\" (UID: \"2df193ce-4a89-4a28-9442-eb9639c22677\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"
Jul 31 23:50:49.482602 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.482557    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/2df193ce-4a89-4a28-9442-eb9639c22677-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-8kq6r\" (UID: \"2df193ce-4a89-4a28-9442-eb9639c22677\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"
Jul 31 23:50:49.520285 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.520263    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:49.583409 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.583383    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/2df193ce-4a89-4a28-9442-eb9639c22677-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-8kq6r\" (UID: \"2df193ce-4a89-4a28-9442-eb9639c22677\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"
Jul 31 23:50:49.583495 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.583423    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4qcm\" (UniqueName: \"kubernetes.io/projected/2df193ce-4a89-4a28-9442-eb9639c22677-kube-api-access-n4qcm\") pod \"keda-metrics-apiserver-55557df8c-8kq6r\" (UID: \"2df193ce-4a89-4a28-9442-eb9639c22677\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"
Jul 31 23:50:49.583495 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.583444    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2df193ce-4a89-4a28-9442-eb9639c22677-certificates\") pod \"keda-metrics-apiserver-55557df8c-8kq6r\" (UID: \"2df193ce-4a89-4a28-9442-eb9639c22677\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"
Jul 31 23:50:49.583563 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:50:49.583540    2546 secret.go:281] references non-existent secret key: tls.crt
Jul 31 23:50:49.583563 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:50:49.583552    2546 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 31 23:50:49.583628 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:50:49.583565    2546 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 31 23:50:49.583628 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:50:49.583580    2546 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 31 23:50:49.583698 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:50:49.583634    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2df193ce-4a89-4a28-9442-eb9639c22677-certificates podName:2df193ce-4a89-4a28-9442-eb9639c22677 nodeName:}" failed. No retries permitted until 2026-07-31 23:50:50.083620707 +0000 UTC m=+280.543083499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2df193ce-4a89-4a28-9442-eb9639c22677-certificates") pod "keda-metrics-apiserver-55557df8c-8kq6r" (UID: "2df193ce-4a89-4a28-9442-eb9639c22677") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 31 23:50:49.583762 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.583746    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/2df193ce-4a89-4a28-9442-eb9639c22677-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-8kq6r\" (UID: \"2df193ce-4a89-4a28-9442-eb9639c22677\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"
Jul 31 23:50:49.591946 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.591930    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4qcm\" (UniqueName: \"kubernetes.io/projected/2df193ce-4a89-4a28-9442-eb9639c22677-kube-api-access-n4qcm\") pod \"keda-metrics-apiserver-55557df8c-8kq6r\" (UID: \"2df193ce-4a89-4a28-9442-eb9639c22677\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"
Jul 31 23:50:49.785818 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.785740    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:49.799039 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.799013    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-vv482"]
Jul 31 23:50:49.802142 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.802128    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-vv482"
Jul 31 23:50:49.804608 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.804591    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 31 23:50:49.812405 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.812383    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-vv482"]
Jul 31 23:50:49.885468 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.885443    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmnk8\" (UniqueName: \"kubernetes.io/projected/3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217-kube-api-access-hmnk8\") pod \"keda-admission-56f9798bb4-vv482\" (UID: \"3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217\") " pod="openshift-keda/keda-admission-56f9798bb4-vv482"
Jul 31 23:50:49.885563 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.885522    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217-certificates\") pod \"keda-admission-56f9798bb4-vv482\" (UID: \"3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217\") " pod="openshift-keda/keda-admission-56f9798bb4-vv482"
Jul 31 23:50:49.899172 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.899143    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:49.986221 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.986201    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217-certificates\") pod \"keda-admission-56f9798bb4-vv482\" (UID: \"3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217\") " pod="openshift-keda/keda-admission-56f9798bb4-vv482"
Jul 31 23:50:49.986556 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:49.986240    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmnk8\" (UniqueName: \"kubernetes.io/projected/3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217-kube-api-access-hmnk8\") pod \"keda-admission-56f9798bb4-vv482\" (UID: \"3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217\") " pod="openshift-keda/keda-admission-56f9798bb4-vv482"
Jul 31 23:50:49.986556 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:50:49.986356    2546 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 31 23:50:49.986556 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:50:49.986377    2546 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-vv482: secret "keda-admission-webhooks-certs" not found
Jul 31 23:50:49.986556 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:50:49.986433    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217-certificates podName:3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217 nodeName:}" failed. No retries permitted until 2026-07-31 23:50:50.486416833 +0000 UTC m=+280.945879640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217-certificates") pod "keda-admission-56f9798bb4-vv482" (UID: "3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217") : secret "keda-admission-webhooks-certs" not found
Jul 31 23:50:50.000062 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:50.000036    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmnk8\" (UniqueName: \"kubernetes.io/projected/3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217-kube-api-access-hmnk8\") pod \"keda-admission-56f9798bb4-vv482\" (UID: \"3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217\") " pod="openshift-keda/keda-admission-56f9798bb4-vv482"
Jul 31 23:50:50.067927 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:50.067859    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:50.086579 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:50.086556    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2df193ce-4a89-4a28-9442-eb9639c22677-certificates\") pod \"keda-metrics-apiserver-55557df8c-8kq6r\" (UID: \"2df193ce-4a89-4a28-9442-eb9639c22677\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"
Jul 31 23:50:50.088915 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:50.088896    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2df193ce-4a89-4a28-9442-eb9639c22677-certificates\") pod \"keda-metrics-apiserver-55557df8c-8kq6r\" (UID: \"2df193ce-4a89-4a28-9442-eb9639c22677\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"
Jul 31 23:50:50.167796 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:50.167769    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:50.251634 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:50.251599    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:50.380226 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:50.380203    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"
Jul 31 23:50:50.488920 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:50.488884    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217-certificates\") pod \"keda-admission-56f9798bb4-vv482\" (UID: \"3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217\") " pod="openshift-keda/keda-admission-56f9798bb4-vv482"
Jul 31 23:50:50.489604 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:50.489572    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:50.491461 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:50.491440    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217-certificates\") pod \"keda-admission-56f9798bb4-vv482\" (UID: \"3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217\") " pod="openshift-keda/keda-admission-56f9798bb4-vv482"
Jul 31 23:50:50.519132 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:50.519109    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"]
Jul 31 23:50:50.521340 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:50:50.521310    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df193ce_4a89_4a28_9442_eb9639c22677.slice/crio-1291645adc36952bee26ecffc7c74ede286e217352dc13257cdd1eecebfaa11c WatchSource:0}: Error finding container 1291645adc36952bee26ecffc7c74ede286e217352dc13257cdd1eecebfaa11c: Status 404 returned error can't find the container with id 1291645adc36952bee26ecffc7c74ede286e217352dc13257cdd1eecebfaa11c
Jul 31 23:50:50.614740 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:50.614709    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:50.712188 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:50.712127    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-vv482"
Jul 31 23:50:50.712922 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:50.712883    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:50:50.842849 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:50.842808    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-vv482"]
Jul 31 23:50:51.383839 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:51.383771    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-vv482" event={"ID":"3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217","Type":"ContainerStarted","Data":"19b50f3b69fbb34292e02c3503cd6c77278655c2913b2c35c280258b3af6e7f9"}
Jul 31 23:50:51.385944 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:51.385879    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r" event={"ID":"2df193ce-4a89-4a28-9442-eb9639c22677","Type":"ContainerStarted","Data":"1291645adc36952bee26ecffc7c74ede286e217352dc13257cdd1eecebfaa11c"}
Jul 31 23:50:54.401514 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:54.401475    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-vv482" event={"ID":"3dae99f7-d3a9-4e5f-8c1f-3d1681c9b217","Type":"ContainerStarted","Data":"ca44727d7ec80351a31dea2e467f1a94de622e696492982eda02dd5a317d3d1c"}
Jul 31 23:50:54.401965 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:54.401623    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-vv482"
Jul 31 23:50:54.402837 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:54.402814    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r" event={"ID":"2df193ce-4a89-4a28-9442-eb9639c22677","Type":"ContainerStarted","Data":"2d69918b13752db985a37b725d1f70d8a494af784a2232e7542571958003d02c"}
Jul 31 23:50:54.402919 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:54.402870    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"
Jul 31 23:50:54.466999 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:54.466956    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-vv482" podStartSLOduration=2.530304659 podStartE2EDuration="5.466944091s" totalImagesPullingTime="2.936639432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:50:54.432574727 +0000 UTC m=+284.892037553" watchObservedRunningTime="2026-07-31 23:50:54.466944091 +0000 UTC m=+284.926406904"
Jul 31 23:50:54.467757 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:50:54.467722    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r" podStartSLOduration=2.2008432 podStartE2EDuration="5.467714147s" totalImagesPullingTime="3.266870947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:50:54.467630047 +0000 UTC m=+284.927092859" watchObservedRunningTime="2026-07-31 23:50:54.467714147 +0000 UTC m=+284.927176960"
Jul 31 23:51:02.375813 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:51:02.375778    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:05.411190 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:51:05.411163    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-8kq6r"
Jul 31 23:51:09.966485 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:51:09.966461    2546 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 23:51:15.409173 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:51:15.409141    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-vv482"
Jul 31 23:51:43.317750 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:51:43.317710    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:51:44.810980 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:51:44.810943    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:11.185657 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.185556    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:11.248411 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.248383    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-j7bld"]
Jul 31 23:52:11.250472 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.250458    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-j7bld"
Jul 31 23:52:11.252854 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.252831    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 31 23:52:11.253938 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.253924    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 31 23:52:11.253938 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.253931    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-k26xl\""
Jul 31 23:52:11.254090 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.253923    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 31 23:52:11.260896 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.260871    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:11.262092 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.262073    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-j7bld"]
Jul 31 23:52:11.302037 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.302011    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f369f278-97a5-4457-b5f1-db1f8aca9940-cert\") pod \"llmisvc-controller-manager-654c557856-j7bld\" (UID: \"f369f278-97a5-4457-b5f1-db1f8aca9940\") " pod="kserve/llmisvc-controller-manager-654c557856-j7bld"
Jul 31 23:52:11.302164 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.302068    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdvkb\" (UniqueName: \"kubernetes.io/projected/f369f278-97a5-4457-b5f1-db1f8aca9940-kube-api-access-cdvkb\") pod \"llmisvc-controller-manager-654c557856-j7bld\" (UID: \"f369f278-97a5-4457-b5f1-db1f8aca9940\") " pod="kserve/llmisvc-controller-manager-654c557856-j7bld"
Jul 31 23:52:11.349476 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.349447    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:11.402802 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.402779    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f369f278-97a5-4457-b5f1-db1f8aca9940-cert\") pod \"llmisvc-controller-manager-654c557856-j7bld\" (UID: \"f369f278-97a5-4457-b5f1-db1f8aca9940\") " pod="kserve/llmisvc-controller-manager-654c557856-j7bld"
Jul 31 23:52:11.402893 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.402839    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvkb\" (UniqueName: \"kubernetes.io/projected/f369f278-97a5-4457-b5f1-db1f8aca9940-kube-api-access-cdvkb\") pod \"llmisvc-controller-manager-654c557856-j7bld\" (UID: \"f369f278-97a5-4457-b5f1-db1f8aca9940\") " pod="kserve/llmisvc-controller-manager-654c557856-j7bld"
Jul 31 23:52:11.402940 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:52:11.402919    2546 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 31 23:52:11.402992 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:52:11.402972    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f369f278-97a5-4457-b5f1-db1f8aca9940-cert podName:f369f278-97a5-4457-b5f1-db1f8aca9940 nodeName:}" failed. No retries permitted until 2026-07-31 23:52:11.902958156 +0000 UTC m=+362.362420947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f369f278-97a5-4457-b5f1-db1f8aca9940-cert") pod "llmisvc-controller-manager-654c557856-j7bld" (UID: "f369f278-97a5-4457-b5f1-db1f8aca9940") : secret "llmisvc-webhook-server-cert" not found
Jul 31 23:52:11.411492 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.411471    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvkb\" (UniqueName: \"kubernetes.io/projected/f369f278-97a5-4457-b5f1-db1f8aca9940-kube-api-access-cdvkb\") pod \"llmisvc-controller-manager-654c557856-j7bld\" (UID: \"f369f278-97a5-4457-b5f1-db1f8aca9940\") " pod="kserve/llmisvc-controller-manager-654c557856-j7bld"
Jul 31 23:52:11.484848 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.484791    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:11.592284 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.592241    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:11.907930 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.907906    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f369f278-97a5-4457-b5f1-db1f8aca9940-cert\") pod \"llmisvc-controller-manager-654c557856-j7bld\" (UID: \"f369f278-97a5-4457-b5f1-db1f8aca9940\") " pod="kserve/llmisvc-controller-manager-654c557856-j7bld"
Jul 31 23:52:11.910236 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:11.910208    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f369f278-97a5-4457-b5f1-db1f8aca9940-cert\") pod \"llmisvc-controller-manager-654c557856-j7bld\" (UID: \"f369f278-97a5-4457-b5f1-db1f8aca9940\") " pod="kserve/llmisvc-controller-manager-654c557856-j7bld"
Jul 31 23:52:12.150166 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:12.150132    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:12.160226 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:12.160164    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-j7bld"
Jul 31 23:52:12.249398 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:12.249364    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:12.298720 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:12.298700    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-j7bld"]
Jul 31 23:52:12.301225 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:52:12.301199    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf369f278_97a5_4457_b5f1_db1f8aca9940.slice/crio-0b7741049d6dc8153ca09e6089c7803c017a146b72fcff2ea1e7fc066ed4c7c7 WatchSource:0}: Error finding container 0b7741049d6dc8153ca09e6089c7803c017a146b72fcff2ea1e7fc066ed4c7c7: Status 404 returned error can't find the container with id 0b7741049d6dc8153ca09e6089c7803c017a146b72fcff2ea1e7fc066ed4c7c7
Jul 31 23:52:12.302516 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:12.302500    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 23:52:12.375037 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:12.375004    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:12.698487 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:12.698457    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-j7bld" event={"ID":"f369f278-97a5-4457-b5f1-db1f8aca9940","Type":"ContainerStarted","Data":"0b7741049d6dc8153ca09e6089c7803c017a146b72fcff2ea1e7fc066ed4c7c7"}
Jul 31 23:52:15.715366 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:15.715329    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-j7bld" event={"ID":"f369f278-97a5-4457-b5f1-db1f8aca9940","Type":"ContainerStarted","Data":"206239a81bda247ec202876ab9387316c945d95a40c192e4192dd6c26fe7deb7"}
Jul 31 23:52:15.715819 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:15.715473    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-j7bld"
Jul 31 23:52:15.732962 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:15.732920    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-j7bld" podStartSLOduration=2.183856157 podStartE2EDuration="4.732905796s" totalImagesPullingTime="2.549049639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:15.732343824 +0000 UTC m=+366.191806637" watchObservedRunningTime="2026-07-31 23:52:15.732905796 +0000 UTC m=+366.192368615"
Jul 31 23:52:44.440035 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.440007    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-776868b596-65p6t"]
Jul 31 23:52:44.443911 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.443887    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.453824 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.453797    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-776868b596-65p6t"]
Jul 31 23:52:44.491870 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.491838    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:44.561243 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.561218    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fce27230-d438-4930-afb1-f0bc442dbbff-service-ca\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.561368 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.561278    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fce27230-d438-4930-afb1-f0bc442dbbff-console-serving-cert\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.561368 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.561316    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fce27230-d438-4930-afb1-f0bc442dbbff-console-config\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.561368 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.561350    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58gk8\" (UniqueName: \"kubernetes.io/projected/fce27230-d438-4930-afb1-f0bc442dbbff-kube-api-access-58gk8\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.561472 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.561437    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fce27230-d438-4930-afb1-f0bc442dbbff-console-oauth-config\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.561472 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.561461    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fce27230-d438-4930-afb1-f0bc442dbbff-trusted-ca-bundle\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.561533 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.561480    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fce27230-d438-4930-afb1-f0bc442dbbff-oauth-serving-cert\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.634905 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.634877    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:44.661951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.661921    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fce27230-d438-4930-afb1-f0bc442dbbff-console-serving-cert\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.662066 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.661952    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fce27230-d438-4930-afb1-f0bc442dbbff-console-config\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.662066 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.661975    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58gk8\" (UniqueName: \"kubernetes.io/projected/fce27230-d438-4930-afb1-f0bc442dbbff-kube-api-access-58gk8\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.662066 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.662019    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fce27230-d438-4930-afb1-f0bc442dbbff-console-oauth-config\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.662066 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.662047    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fce27230-d438-4930-afb1-f0bc442dbbff-trusted-ca-bundle\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.662306 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.662077    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fce27230-d438-4930-afb1-f0bc442dbbff-oauth-serving-cert\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.662306 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.662133    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fce27230-d438-4930-afb1-f0bc442dbbff-service-ca\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.662773 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.662753    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fce27230-d438-4930-afb1-f0bc442dbbff-console-config\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.662932 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.662905    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fce27230-d438-4930-afb1-f0bc442dbbff-oauth-serving-cert\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.663604 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.663580    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fce27230-d438-4930-afb1-f0bc442dbbff-trusted-ca-bundle\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.664403 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.664383    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fce27230-d438-4930-afb1-f0bc442dbbff-console-oauth-config\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.664634 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.664616    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fce27230-d438-4930-afb1-f0bc442dbbff-console-serving-cert\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.665804 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.665786    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fce27230-d438-4930-afb1-f0bc442dbbff-service-ca\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.670381 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.670360    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58gk8\" (UniqueName: \"kubernetes.io/projected/fce27230-d438-4930-afb1-f0bc442dbbff-kube-api-access-58gk8\") pod \"console-776868b596-65p6t\" (UID: \"fce27230-d438-4930-afb1-f0bc442dbbff\") " pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.690704 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.690646    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:44.752751 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.752724    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:44.902402 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:44.902370    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-776868b596-65p6t"]
Jul 31 23:52:44.905418 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:52:44.905391    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce27230_d438_4930_afb1_f0bc442dbbff.slice/crio-dcf32241e613bff03e36deaf87cdfd34bbe0f85d4e209627af26cc6ced041d6c WatchSource:0}: Error finding container dcf32241e613bff03e36deaf87cdfd34bbe0f85d4e209627af26cc6ced041d6c: Status 404 returned error can't find the container with id dcf32241e613bff03e36deaf87cdfd34bbe0f85d4e209627af26cc6ced041d6c
Jul 31 23:52:45.235542 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:45.235459    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:45.293112 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:45.293078    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:45.831564 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:45.831531    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776868b596-65p6t" event={"ID":"fce27230-d438-4930-afb1-f0bc442dbbff","Type":"ContainerStarted","Data":"dc5e084de835bfc627770b1d5da69a61e047f1dfed38ab64753f4db2c1342d35"}
Jul 31 23:52:45.831919 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:45.831578    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776868b596-65p6t" event={"ID":"fce27230-d438-4930-afb1-f0bc442dbbff","Type":"ContainerStarted","Data":"dcf32241e613bff03e36deaf87cdfd34bbe0f85d4e209627af26cc6ced041d6c"}
Jul 31 23:52:45.851261 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:45.851204    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-776868b596-65p6t" podStartSLOduration=1.8511880600000001 podStartE2EDuration="1.85118806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:52:45.849514284 +0000 UTC m=+396.308977097" watchObservedRunningTime="2026-07-31 23:52:45.85118806 +0000 UTC m=+396.310650873"
Jul 31 23:52:46.722059 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:46.722032    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-j7bld"
Jul 31 23:52:54.753502 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:54.753468    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:54.753894 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:54.753516    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:54.758290 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:54.758245    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:54.869145 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:54.869117    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-776868b596-65p6t"
Jul 31 23:52:54.914713 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:54.914686    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f8f79dcf4-qxj2w"]
Jul 31 23:52:55.028996 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:55.028920    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:55.105200 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:55.105168    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:52:55.161710 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:52:55.161677    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:19.934149 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:19.934087    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f8f79dcf4-qxj2w" podUID="50cbd0c3-2433-4666-8e6c-d33e505ec692" containerName="console" containerID="cri-o://736fd1c93eafeee40136ee516750d8eb145888924cfba84ced955d4acea28606" gracePeriod=15
Jul 31 23:53:20.179429 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.179404    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:53:20.239656 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.239590    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-oauth-serving-cert\") pod \"50cbd0c3-2433-4666-8e6c-d33e505ec692\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") "
Jul 31 23:53:20.239656 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.239634    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-serving-cert\") pod \"50cbd0c3-2433-4666-8e6c-d33e505ec692\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") "
Jul 31 23:53:20.239847 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.239683    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-config\") pod \"50cbd0c3-2433-4666-8e6c-d33e505ec692\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") "
Jul 31 23:53:20.239847 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.239707    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-service-ca\") pod \"50cbd0c3-2433-4666-8e6c-d33e505ec692\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") "
Jul 31 23:53:20.239847 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.239724    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-trusted-ca-bundle\") pod \"50cbd0c3-2433-4666-8e6c-d33e505ec692\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") "
Jul 31 23:53:20.239847 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.239744    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qnbn\" (UniqueName: \"kubernetes.io/projected/50cbd0c3-2433-4666-8e6c-d33e505ec692-kube-api-access-6qnbn\") pod \"50cbd0c3-2433-4666-8e6c-d33e505ec692\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") "
Jul 31 23:53:20.239847 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.239772    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-oauth-config\") pod \"50cbd0c3-2433-4666-8e6c-d33e505ec692\" (UID: \"50cbd0c3-2433-4666-8e6c-d33e505ec692\") "
Jul 31 23:53:20.240095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.240070    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "50cbd0c3-2433-4666-8e6c-d33e505ec692" (UID: "50cbd0c3-2433-4666-8e6c-d33e505ec692"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:53:20.240168 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.240142    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-config" (OuterVolumeSpecName: "console-config") pod "50cbd0c3-2433-4666-8e6c-d33e505ec692" (UID: "50cbd0c3-2433-4666-8e6c-d33e505ec692"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:53:20.240269 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.240165    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-service-ca" (OuterVolumeSpecName: "service-ca") pod "50cbd0c3-2433-4666-8e6c-d33e505ec692" (UID: "50cbd0c3-2433-4666-8e6c-d33e505ec692"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:53:20.240269 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.240168    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "50cbd0c3-2433-4666-8e6c-d33e505ec692" (UID: "50cbd0c3-2433-4666-8e6c-d33e505ec692"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:53:20.241853 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.241829    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "50cbd0c3-2433-4666-8e6c-d33e505ec692" (UID: "50cbd0c3-2433-4666-8e6c-d33e505ec692"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:53:20.241937 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.241884    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cbd0c3-2433-4666-8e6c-d33e505ec692-kube-api-access-6qnbn" (OuterVolumeSpecName: "kube-api-access-6qnbn") pod "50cbd0c3-2433-4666-8e6c-d33e505ec692" (UID: "50cbd0c3-2433-4666-8e6c-d33e505ec692"). InnerVolumeSpecName "kube-api-access-6qnbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:53:20.241979 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.241930    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "50cbd0c3-2433-4666-8e6c-d33e505ec692" (UID: "50cbd0c3-2433-4666-8e6c-d33e505ec692"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:53:20.341134 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.341110    2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-oauth-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:53:20.341134 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.341131    2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-oauth-serving-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:53:20.341300 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.341142    2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-serving-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:53:20.341300 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.341151    2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-console-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:53:20.341300 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.341160    2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-service-ca\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:53:20.341300 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.341169    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50cbd0c3-2433-4666-8e6c-d33e505ec692-trusted-ca-bundle\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:53:20.341300 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.341177    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qnbn\" (UniqueName: \"kubernetes.io/projected/50cbd0c3-2433-4666-8e6c-d33e505ec692-kube-api-access-6qnbn\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:53:20.971930 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.971894    2546 generic.go:356] "Generic (PLEG): container finished" podID="50cbd0c3-2433-4666-8e6c-d33e505ec692" containerID="736fd1c93eafeee40136ee516750d8eb145888924cfba84ced955d4acea28606" exitCode=0
Jul 31 23:53:20.972334 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.971942    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8f79dcf4-qxj2w"
Jul 31 23:53:20.972334 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.971977    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8f79dcf4-qxj2w" event={"ID":"50cbd0c3-2433-4666-8e6c-d33e505ec692","Type":"ContainerDied","Data":"736fd1c93eafeee40136ee516750d8eb145888924cfba84ced955d4acea28606"}
Jul 31 23:53:20.972334 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.972043    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8f79dcf4-qxj2w" event={"ID":"50cbd0c3-2433-4666-8e6c-d33e505ec692","Type":"ContainerDied","Data":"787f35b3a3be73d28730c0ee5e40c502db0fe56d99693375ab3aeb14590700d7"}
Jul 31 23:53:20.972334 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.972064    2546 scope.go:117] "RemoveContainer" containerID="736fd1c93eafeee40136ee516750d8eb145888924cfba84ced955d4acea28606"
Jul 31 23:53:20.980541 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.980522    2546 scope.go:117] "RemoveContainer" containerID="736fd1c93eafeee40136ee516750d8eb145888924cfba84ced955d4acea28606"
Jul 31 23:53:20.980808 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:53:20.980789    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736fd1c93eafeee40136ee516750d8eb145888924cfba84ced955d4acea28606\": container with ID starting with 736fd1c93eafeee40136ee516750d8eb145888924cfba84ced955d4acea28606 not found: ID does not exist" containerID="736fd1c93eafeee40136ee516750d8eb145888924cfba84ced955d4acea28606"
Jul 31 23:53:20.980854 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.980816    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736fd1c93eafeee40136ee516750d8eb145888924cfba84ced955d4acea28606"} err="failed to get container status \"736fd1c93eafeee40136ee516750d8eb145888924cfba84ced955d4acea28606\": rpc error: code = NotFound desc = could not find container \"736fd1c93eafeee40136ee516750d8eb145888924cfba84ced955d4acea28606\": container with ID starting with 736fd1c93eafeee40136ee516750d8eb145888924cfba84ced955d4acea28606 not found: ID does not exist"
Jul 31 23:53:20.993308 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.993290    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f8f79dcf4-qxj2w"]
Jul 31 23:53:20.995799 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:20.995782    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f8f79dcf4-qxj2w"]
Jul 31 23:53:22.075192 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:22.075159    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50cbd0c3-2433-4666-8e6c-d33e505ec692" path="/var/lib/kubelet/pods/50cbd0c3-2433-4666-8e6c-d33e505ec692/volumes"
Jul 31 23:53:28.177008 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.176972    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:28.219161 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.219134    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-g74kd"]
Jul 31 23:53:28.219475 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.219463    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50cbd0c3-2433-4666-8e6c-d33e505ec692" containerName="console"
Jul 31 23:53:28.219521 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.219476    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="50cbd0c3-2433-4666-8e6c-d33e505ec692" containerName="console"
Jul 31 23:53:28.219554 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.219536    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="50cbd0c3-2433-4666-8e6c-d33e505ec692" containerName="console"
Jul 31 23:53:28.222147 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.222130    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-g74kd"
Jul 31 23:53:28.224503 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.224475    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 31 23:53:28.224597 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.224580    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-nlldr\""
Jul 31 23:53:28.232450 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.232431    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-g74kd"]
Jul 31 23:53:28.263761 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.263714    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:28.297675 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.297650    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs99f\" (UniqueName: \"kubernetes.io/projected/ed88cc15-1b2d-4554-93e2-5b8b01298e6b-kube-api-access-fs99f\") pod \"model-serving-api-7b954ffc95-g74kd\" (UID: \"ed88cc15-1b2d-4554-93e2-5b8b01298e6b\") " pod="kserve/model-serving-api-7b954ffc95-g74kd"
Jul 31 23:53:28.297755 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.297681    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed88cc15-1b2d-4554-93e2-5b8b01298e6b-tls-certs\") pod \"model-serving-api-7b954ffc95-g74kd\" (UID: \"ed88cc15-1b2d-4554-93e2-5b8b01298e6b\") " pod="kserve/model-serving-api-7b954ffc95-g74kd"
Jul 31 23:53:28.319947 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.319921    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:28.398186 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.398157    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs99f\" (UniqueName: \"kubernetes.io/projected/ed88cc15-1b2d-4554-93e2-5b8b01298e6b-kube-api-access-fs99f\") pod \"model-serving-api-7b954ffc95-g74kd\" (UID: \"ed88cc15-1b2d-4554-93e2-5b8b01298e6b\") " pod="kserve/model-serving-api-7b954ffc95-g74kd"
Jul 31 23:53:28.398295 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.398191    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed88cc15-1b2d-4554-93e2-5b8b01298e6b-tls-certs\") pod \"model-serving-api-7b954ffc95-g74kd\" (UID: \"ed88cc15-1b2d-4554-93e2-5b8b01298e6b\") " pod="kserve/model-serving-api-7b954ffc95-g74kd"
Jul 31 23:53:28.400324 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.400307    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed88cc15-1b2d-4554-93e2-5b8b01298e6b-tls-certs\") pod \"model-serving-api-7b954ffc95-g74kd\" (UID: \"ed88cc15-1b2d-4554-93e2-5b8b01298e6b\") " pod="kserve/model-serving-api-7b954ffc95-g74kd"
Jul 31 23:53:28.405503 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.405485    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs99f\" (UniqueName: \"kubernetes.io/projected/ed88cc15-1b2d-4554-93e2-5b8b01298e6b-kube-api-access-fs99f\") pod \"model-serving-api-7b954ffc95-g74kd\" (UID: \"ed88cc15-1b2d-4554-93e2-5b8b01298e6b\") " pod="kserve/model-serving-api-7b954ffc95-g74kd"
Jul 31 23:53:28.420210 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.420186    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:28.532743 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.532695    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-g74kd"
Jul 31 23:53:28.658421 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.658379    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-g74kd"]
Jul 31 23:53:28.660551 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:53:28.660525    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded88cc15_1b2d_4554_93e2_5b8b01298e6b.slice/crio-d4328053f54a50cc25338baaaa72aaf80d87e6934a72f3574b9d8ce3006e1c6a WatchSource:0}: Error finding container d4328053f54a50cc25338baaaa72aaf80d87e6934a72f3574b9d8ce3006e1c6a: Status 404 returned error can't find the container with id d4328053f54a50cc25338baaaa72aaf80d87e6934a72f3574b9d8ce3006e1c6a
Jul 31 23:53:28.898019 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.897981    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:28.952629 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:28.952601    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:53:29.006345 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:29.006320    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-g74kd" event={"ID":"ed88cc15-1b2d-4554-93e2-5b8b01298e6b","Type":"ContainerStarted","Data":"d4328053f54a50cc25338baaaa72aaf80d87e6934a72f3574b9d8ce3006e1c6a"}
Jul 31 23:53:31.020468 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:31.020430    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-g74kd" event={"ID":"ed88cc15-1b2d-4554-93e2-5b8b01298e6b","Type":"ContainerStarted","Data":"f967e18751167e71dd603c2bca89924b10b483ee14027fc8a5d15e18c07c9c63"}
Jul 31 23:53:31.020827 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:31.020597    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-g74kd"
Jul 31 23:53:31.039288 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:31.039231    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-g74kd" podStartSLOduration=0.874895135 podStartE2EDuration="3.039217435s" totalImagesPullingTime="2.1643223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:53:31.037471646 +0000 UTC m=+441.496934460" watchObservedRunningTime="2026-07-31 23:53:31.039217435 +0000 UTC m=+441.498680248"
Jul 31 23:53:42.028632 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:53:42.028604    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-g74kd"
Jul 31 23:55:01.563154 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:01.563114    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"]
Jul 31 23:55:01.565725 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:01.565708    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"
Jul 31 23:55:01.568106 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:01.568075    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 31 23:55:01.568227 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:01.568153    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-rvpq8\""
Jul 31 23:55:01.574380 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:01.574359    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"]
Jul 31 23:55:01.598398 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:01.598376    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:01.694785 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:01.694759    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbjq5\" (UniqueName: \"kubernetes.io/projected/7008fa62-8ade-4665-a06d-b7194913ceeb-kube-api-access-mbjq5\") pod \"seaweedfs-tls-custom-6644bf5d6c-ldq4l\" (UID: \"7008fa62-8ade-4665-a06d-b7194913ceeb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"
Jul 31 23:55:01.694871 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:01.694823    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7008fa62-8ade-4665-a06d-b7194913ceeb-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-ldq4l\" (UID: \"7008fa62-8ade-4665-a06d-b7194913ceeb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"
Jul 31 23:55:01.795783 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:01.795754    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbjq5\" (UniqueName: \"kubernetes.io/projected/7008fa62-8ade-4665-a06d-b7194913ceeb-kube-api-access-mbjq5\") pod \"seaweedfs-tls-custom-6644bf5d6c-ldq4l\" (UID: \"7008fa62-8ade-4665-a06d-b7194913ceeb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"
Jul 31 23:55:01.795938 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:01.795815    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7008fa62-8ade-4665-a06d-b7194913ceeb-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-ldq4l\" (UID: \"7008fa62-8ade-4665-a06d-b7194913ceeb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"
Jul 31 23:55:01.796151 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:01.796135    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7008fa62-8ade-4665-a06d-b7194913ceeb-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-ldq4l\" (UID: \"7008fa62-8ade-4665-a06d-b7194913ceeb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"
Jul 31 23:55:01.804538 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:01.804508    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbjq5\" (UniqueName: \"kubernetes.io/projected/7008fa62-8ade-4665-a06d-b7194913ceeb-kube-api-access-mbjq5\") pod \"seaweedfs-tls-custom-6644bf5d6c-ldq4l\" (UID: \"7008fa62-8ade-4665-a06d-b7194913ceeb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"
Jul 31 23:55:01.874911 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:01.874885    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"
Jul 31 23:55:01.999342 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:01.999317    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"]
Jul 31 23:55:02.001893 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:55:02.001868    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7008fa62_8ade_4665_a06d_b7194913ceeb.slice/crio-b75f5452cfc2534321dbe8bd265a432e669c35fd758a1acdf91d9890d0ac8041 WatchSource:0}: Error finding container b75f5452cfc2534321dbe8bd265a432e669c35fd758a1acdf91d9890d0ac8041: Status 404 returned error can't find the container with id b75f5452cfc2534321dbe8bd265a432e669c35fd758a1acdf91d9890d0ac8041
Jul 31 23:55:02.364115 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:02.364080    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l" event={"ID":"7008fa62-8ade-4665-a06d-b7194913ceeb","Type":"ContainerStarted","Data":"b75f5452cfc2534321dbe8bd265a432e669c35fd758a1acdf91d9890d0ac8041"}
Jul 31 23:55:05.377421 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:05.377380    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l" event={"ID":"7008fa62-8ade-4665-a06d-b7194913ceeb","Type":"ContainerStarted","Data":"bcc809044de1c4624e0cb1c641d8ea95f85640809f8050abeaa7a5d9398d5976"}
Jul 31 23:55:05.392482 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:05.392437    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l" podStartSLOduration=1.7326585159999999 podStartE2EDuration="4.392421434s" totalImagesPullingTime="2.659762918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:55:05.391888884 +0000 UTC m=+535.851351698" watchObservedRunningTime="2026-07-31 23:55:05.392421434 +0000 UTC m=+535.851884246"
Jul 31 23:55:06.300192 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:06.300162    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"]
Jul 31 23:55:07.384801 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:07.384763    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l" podUID="7008fa62-8ade-4665-a06d-b7194913ceeb" containerName="seaweedfs-tls-custom" containerID="cri-o://bcc809044de1c4624e0cb1c641d8ea95f85640809f8050abeaa7a5d9398d5976" gracePeriod=30
Jul 31 23:55:12.480856 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:12.480824    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:35.319990 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.319965    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"
Jul 31 23:55:35.361000 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.360972    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7008fa62-8ade-4665-a06d-b7194913ceeb-data\") pod \"7008fa62-8ade-4665-a06d-b7194913ceeb\" (UID: \"7008fa62-8ade-4665-a06d-b7194913ceeb\") "
Jul 31 23:55:35.361158 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.361070    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbjq5\" (UniqueName: \"kubernetes.io/projected/7008fa62-8ade-4665-a06d-b7194913ceeb-kube-api-access-mbjq5\") pod \"7008fa62-8ade-4665-a06d-b7194913ceeb\" (UID: \"7008fa62-8ade-4665-a06d-b7194913ceeb\") "
Jul 31 23:55:35.362341 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.362307    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7008fa62-8ade-4665-a06d-b7194913ceeb-data" (OuterVolumeSpecName: "data") pod "7008fa62-8ade-4665-a06d-b7194913ceeb" (UID: "7008fa62-8ade-4665-a06d-b7194913ceeb"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:55:35.363462 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.363438    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7008fa62-8ade-4665-a06d-b7194913ceeb-kube-api-access-mbjq5" (OuterVolumeSpecName: "kube-api-access-mbjq5") pod "7008fa62-8ade-4665-a06d-b7194913ceeb" (UID: "7008fa62-8ade-4665-a06d-b7194913ceeb"). InnerVolumeSpecName "kube-api-access-mbjq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:55:35.462393 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.462315    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbjq5\" (UniqueName: \"kubernetes.io/projected/7008fa62-8ade-4665-a06d-b7194913ceeb-kube-api-access-mbjq5\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:55:35.462393 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.462341    2546 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7008fa62-8ade-4665-a06d-b7194913ceeb-data\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:55:35.501799 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.501774    2546 generic.go:356] "Generic (PLEG): container finished" podID="7008fa62-8ade-4665-a06d-b7194913ceeb" containerID="bcc809044de1c4624e0cb1c641d8ea95f85640809f8050abeaa7a5d9398d5976" exitCode=0
Jul 31 23:55:35.501938 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.501816    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"
Jul 31 23:55:35.501938 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.501847    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l" event={"ID":"7008fa62-8ade-4665-a06d-b7194913ceeb","Type":"ContainerDied","Data":"bcc809044de1c4624e0cb1c641d8ea95f85640809f8050abeaa7a5d9398d5976"}
Jul 31 23:55:35.501938 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.501884    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l" event={"ID":"7008fa62-8ade-4665-a06d-b7194913ceeb","Type":"ContainerDied","Data":"b75f5452cfc2534321dbe8bd265a432e669c35fd758a1acdf91d9890d0ac8041"}
Jul 31 23:55:35.501938 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.501904    2546 scope.go:117] "RemoveContainer" containerID="bcc809044de1c4624e0cb1c641d8ea95f85640809f8050abeaa7a5d9398d5976"
Jul 31 23:55:35.514345 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.514320    2546 scope.go:117] "RemoveContainer" containerID="bcc809044de1c4624e0cb1c641d8ea95f85640809f8050abeaa7a5d9398d5976"
Jul 31 23:55:35.514782 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:55:35.514762    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc809044de1c4624e0cb1c641d8ea95f85640809f8050abeaa7a5d9398d5976\": container with ID starting with bcc809044de1c4624e0cb1c641d8ea95f85640809f8050abeaa7a5d9398d5976 not found: ID does not exist" containerID="bcc809044de1c4624e0cb1c641d8ea95f85640809f8050abeaa7a5d9398d5976"
Jul 31 23:55:35.514861 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.514794    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc809044de1c4624e0cb1c641d8ea95f85640809f8050abeaa7a5d9398d5976"} err="failed to get container status \"bcc809044de1c4624e0cb1c641d8ea95f85640809f8050abeaa7a5d9398d5976\": rpc error: code = NotFound desc = could not find container \"bcc809044de1c4624e0cb1c641d8ea95f85640809f8050abeaa7a5d9398d5976\": container with ID starting with bcc809044de1c4624e0cb1c641d8ea95f85640809f8050abeaa7a5d9398d5976 not found: ID does not exist"
Jul 31 23:55:35.523987 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.523964    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"]
Jul 31 23:55:35.527872 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:35.527853    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ldq4l"]
Jul 31 23:55:36.075754 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:36.075722    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7008fa62-8ade-4665-a06d-b7194913ceeb" path="/var/lib/kubelet/pods/7008fa62-8ade-4665-a06d-b7194913ceeb/volumes"
Jul 31 23:55:48.595483 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:48.595449    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:48.919864 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:48.919829    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:55:58.173308 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:55:58.173268    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:03.426299 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:03.426224    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:03.515160 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:03.515133    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:14.820135 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.820094    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:14.878483 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.878447    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"]
Jul 31 23:56:14.878982 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.878962    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7008fa62-8ade-4665-a06d-b7194913ceeb" containerName="seaweedfs-tls-custom"
Jul 31 23:56:14.879026 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.878987    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7008fa62-8ade-4665-a06d-b7194913ceeb" containerName="seaweedfs-tls-custom"
Jul 31 23:56:14.879077 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.879065    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7008fa62-8ade-4665-a06d-b7194913ceeb" containerName="seaweedfs-tls-custom"
Jul 31 23:56:14.881748 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.881733    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:14.884454 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.884425    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 31 23:56:14.884584 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.884430    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 23:56:14.884584 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.884542    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 31 23:56:14.884584 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.884554    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 23:56:14.884756 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.884743    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 23:56:14.884827 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.884811    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Jul 31 23:56:14.884895 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.884859    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 23:56:14.891864 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.891835    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"]
Jul 31 23:56:14.963508 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.963474    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvfj7\" (UniqueName: \"kubernetes.io/projected/8584ee88-5809-4014-87d7-6cfdc124f7c0-kube-api-access-kvfj7\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:14.963816 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.963784    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8584ee88-5809-4014-87d7-6cfdc124f7c0-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:14.963913 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.963849    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8584ee88-5809-4014-87d7-6cfdc124f7c0-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:14.963954 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.963914    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8584ee88-5809-4014-87d7-6cfdc124f7c0-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:14.963954 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:14.963945    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8584ee88-5809-4014-87d7-6cfdc124f7c0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:15.006957 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.006924    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:15.065387 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.065344    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8584ee88-5809-4014-87d7-6cfdc124f7c0-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:15.065563 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.065396    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8584ee88-5809-4014-87d7-6cfdc124f7c0-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:15.065563 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.065439    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8584ee88-5809-4014-87d7-6cfdc124f7c0-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:15.065563 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.065459    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8584ee88-5809-4014-87d7-6cfdc124f7c0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:15.065563 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.065486    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvfj7\" (UniqueName: \"kubernetes.io/projected/8584ee88-5809-4014-87d7-6cfdc124f7c0-kube-api-access-kvfj7\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:15.065756 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:56:15.065613    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 31 23:56:15.065756 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:56:15.065698    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8584ee88-5809-4014-87d7-6cfdc124f7c0-proxy-tls podName:8584ee88-5809-4014-87d7-6cfdc124f7c0 nodeName:}" failed. No retries permitted until 2026-07-31 23:56:15.565674824 +0000 UTC m=+606.025137633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8584ee88-5809-4014-87d7-6cfdc124f7c0-proxy-tls") pod "isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" (UID: "8584ee88-5809-4014-87d7-6cfdc124f7c0") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 31 23:56:15.065842 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.065799    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8584ee88-5809-4014-87d7-6cfdc124f7c0-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:15.066146 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.066127    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8584ee88-5809-4014-87d7-6cfdc124f7c0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:15.066187 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.066127    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8584ee88-5809-4014-87d7-6cfdc124f7c0-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:15.076162 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.076086    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvfj7\" (UniqueName: \"kubernetes.io/projected/8584ee88-5809-4014-87d7-6cfdc124f7c0-kube-api-access-kvfj7\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:15.106823 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.106784    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:15.346025 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.345945    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:15.570072 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.569962    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8584ee88-5809-4014-87d7-6cfdc124f7c0-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:15.575596 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.572361    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8584ee88-5809-4014-87d7-6cfdc124f7c0-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7f9c4f966-f7v69\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:15.794297 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.794235    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:15.929119 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:15.929088    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"]
Jul 31 23:56:15.931461 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:56:15.931435    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8584ee88_5809_4014_87d7_6cfdc124f7c0.slice/crio-217ab0a92773deffd6eef105954c39d3450e437de19a6378e6665319d75cff1c WatchSource:0}: Error finding container 217ab0a92773deffd6eef105954c39d3450e437de19a6378e6665319d75cff1c: Status 404 returned error can't find the container with id 217ab0a92773deffd6eef105954c39d3450e437de19a6378e6665319d75cff1c
Jul 31 23:56:16.669064 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:16.669028    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" event={"ID":"8584ee88-5809-4014-87d7-6cfdc124f7c0","Type":"ContainerStarted","Data":"217ab0a92773deffd6eef105954c39d3450e437de19a6378e6665319d75cff1c"}
Jul 31 23:56:19.682588 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:19.682550    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" event={"ID":"8584ee88-5809-4014-87d7-6cfdc124f7c0","Type":"ContainerStarted","Data":"3ee32e0e1f9928ab16f82754266edef28f338c98c374e0592613c1cf3ff02a75"}
Jul 31 23:56:22.695638 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:22.695561    2546 generic.go:356] "Generic (PLEG): container finished" podID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerID="3ee32e0e1f9928ab16f82754266edef28f338c98c374e0592613c1cf3ff02a75" exitCode=0
Jul 31 23:56:22.695638 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:22.695619    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" event={"ID":"8584ee88-5809-4014-87d7-6cfdc124f7c0","Type":"ContainerDied","Data":"3ee32e0e1f9928ab16f82754266edef28f338c98c374e0592613c1cf3ff02a75"}
Jul 31 23:56:36.767514 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:36.767476    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" event={"ID":"8584ee88-5809-4014-87d7-6cfdc124f7c0","Type":"ContainerStarted","Data":"413da9d06ab1b5f07eed9744287e3f231efe594118a34a193de689b20a105182"}
Jul 31 23:56:38.780207 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:38.780167    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" event={"ID":"8584ee88-5809-4014-87d7-6cfdc124f7c0","Type":"ContainerStarted","Data":"a488a9fba1b861df9eed0ff8de0468a1464ddbff83c380111ea0d02229bf6c88"}
Jul 31 23:56:40.792164 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:40.792129    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" event={"ID":"8584ee88-5809-4014-87d7-6cfdc124f7c0","Type":"ContainerStarted","Data":"2111257af11dd3bd5b0c71250d6ea010d85e1a8dcbc69043ad82ea846ea9c84b"}
Jul 31 23:56:40.792567 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:40.792358    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:40.814522 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:40.814474    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podStartSLOduration=3.136680668 podStartE2EDuration="26.81445857s" totalImagesPullingTime="20.032721499s" totalInitContainerRuntime="3.645056403s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:56:14 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:56:40.811968729 +0000 UTC m=+631.271431541" watchObservedRunningTime="2026-07-31 23:56:40.81445857 +0000 UTC m=+631.273921382"
Jul 31 23:56:41.795695 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:41.795658    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:41.796142 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:41.795762    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:41.797336 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:41.797297    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:56:41.798274 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:41.798227    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:56:42.800070 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:42.800027    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:56:42.800514 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:42.800452    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:56:47.804902 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:47.804865    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:56:47.805527 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:47.805496    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:56:47.805930 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:47.805906    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:56:54.879140 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:54.879110    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:56:57.806060 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:57.806023    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:56:57.806481 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:56:57.806453    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:57:07.805842 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:07.805800    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:57:07.806359 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:07.806281    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:57:17.806139 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:17.806091    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:57:17.806587 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:17.806562    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:57:27.806331 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:27.806285    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:57:27.806680 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:27.806657    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:57:37.806270 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:37.806224    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:57:37.806760 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:37.806735    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:57:47.806483 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:47.806450    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:57:47.806924 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:47.806692    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:57:56.106687 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:56.106657    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:57:59.950982 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:59.950941    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"]
Jul 31 23:57:59.951363 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:59.951334    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kserve-container" containerID="cri-o://413da9d06ab1b5f07eed9744287e3f231efe594118a34a193de689b20a105182" gracePeriod=30
Jul 31 23:57:59.951363 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:59.951343    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kube-rbac-proxy" containerID="cri-o://a488a9fba1b861df9eed0ff8de0468a1464ddbff83c380111ea0d02229bf6c88" gracePeriod=30
Jul 31 23:57:59.951484 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:59.951428    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="agent" containerID="cri-o://2111257af11dd3bd5b0c71250d6ea010d85e1a8dcbc69043ad82ea846ea9c84b" gracePeriod=30
Jul 31 23:57:59.967803 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:57:59.967776    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:00.060463 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.060425    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:00.064453 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.064427    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"]
Jul 31 23:58:00.067059 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.067043    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.069476 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.069455    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 31 23:58:00.069577 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.069561    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 31 23:58:00.078885 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.078864    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"]
Jul 31 23:58:00.100411 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.100392    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f88fbfdf-09ad-408a-87a5-e674fc64b373-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.100508 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.100432    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwtcb\" (UniqueName: \"kubernetes.io/projected/f88fbfdf-09ad-408a-87a5-e674fc64b373-kube-api-access-dwtcb\") pod \"isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.100564 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.100514    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88fbfdf-09ad-408a-87a5-e674fc64b373-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.100619 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.100607    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f88fbfdf-09ad-408a-87a5-e674fc64b373-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.100678 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.100644    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f88fbfdf-09ad-408a-87a5-e674fc64b373-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.114554 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.114531    2546 generic.go:356] "Generic (PLEG): container finished" podID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerID="a488a9fba1b861df9eed0ff8de0468a1464ddbff83c380111ea0d02229bf6c88" exitCode=2
Jul 31 23:58:00.114633 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.114559    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" event={"ID":"8584ee88-5809-4014-87d7-6cfdc124f7c0","Type":"ContainerDied","Data":"a488a9fba1b861df9eed0ff8de0468a1464ddbff83c380111ea0d02229bf6c88"}
Jul 31 23:58:00.201900 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.201837    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88fbfdf-09ad-408a-87a5-e674fc64b373-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.201900 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.201867    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f88fbfdf-09ad-408a-87a5-e674fc64b373-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.201900 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.201899    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f88fbfdf-09ad-408a-87a5-e674fc64b373-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.202102 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.201920    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f88fbfdf-09ad-408a-87a5-e674fc64b373-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.202102 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.201963    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwtcb\" (UniqueName: \"kubernetes.io/projected/f88fbfdf-09ad-408a-87a5-e674fc64b373-kube-api-access-dwtcb\") pod \"isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.202307 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.202288    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f88fbfdf-09ad-408a-87a5-e674fc64b373-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.202608 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.202587    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f88fbfdf-09ad-408a-87a5-e674fc64b373-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.202670 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.202595    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f88fbfdf-09ad-408a-87a5-e674fc64b373-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.204240 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.204218    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88fbfdf-09ad-408a-87a5-e674fc64b373-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.209455 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.209429    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:00.210795 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.210776    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwtcb\" (UniqueName: \"kubernetes.io/projected/f88fbfdf-09ad-408a-87a5-e674fc64b373-kube-api-access-dwtcb\") pod \"isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.376490 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.376447    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:00.507382 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:00.507363    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"]
Jul 31 23:58:00.509650 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:58:00.509620    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88fbfdf_09ad_408a_87a5_e674fc64b373.slice/crio-6a6cb47ee8864f191f77979132b2430490db8d621741b4a149c6c5f23e656845 WatchSource:0}: Error finding container 6a6cb47ee8864f191f77979132b2430490db8d621741b4a149c6c5f23e656845: Status 404 returned error can't find the container with id 6a6cb47ee8864f191f77979132b2430490db8d621741b4a149c6c5f23e656845
Jul 31 23:58:01.120403 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.120364    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" event={"ID":"f88fbfdf-09ad-408a-87a5-e674fc64b373","Type":"ContainerStarted","Data":"ce077cd9f6d05adeaab196fe4c1beecbcb7ca6f84765399653ab46a106541d0a"}
Jul 31 23:58:01.120781 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.120448    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" event={"ID":"f88fbfdf-09ad-408a-87a5-e674fc64b373","Type":"ContainerStarted","Data":"6a6cb47ee8864f191f77979132b2430490db8d621741b4a149c6c5f23e656845"}
Jul 31 23:58:01.187846 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.187813    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:01.221542 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.221509    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"]
Jul 31 23:58:01.224785 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.224766    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.227315 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.227296    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-cfe5a5-predictor-serving-cert\""
Jul 31 23:58:01.227514 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.227498    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-cfe5a5-kube-rbac-proxy-sar-config\""
Jul 31 23:58:01.233627 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.233608    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"]
Jul 31 23:58:01.310382 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.310359    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-cabundle-cert\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.310483 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.310398    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jmsq\" (UniqueName: \"kubernetes.io/projected/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kube-api-access-8jmsq\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.310483 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.310419    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-proxy-tls\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.310578 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.310493    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.310578 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.310526    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kserve-provision-location\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.411813 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.411759    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-cabundle-cert\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.411813 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.411798    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jmsq\" (UniqueName: \"kubernetes.io/projected/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kube-api-access-8jmsq\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.411994 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.411815    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-proxy-tls\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.411994 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.411860    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.411994 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:58:01.411940    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-serving-cert: secret "isvc-primary-cfe5a5-predictor-serving-cert" not found
Jul 31 23:58:01.411994 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.411976    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kserve-provision-location\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.411994 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:58:01.411994    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-proxy-tls podName:ec5fabb1-b92b-4ef4-b966-b50a5203b07a nodeName:}" failed. No retries permitted until 2026-07-31 23:58:01.911976135 +0000 UTC m=+712.371438929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-proxy-tls") pod "isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" (UID: "ec5fabb1-b92b-4ef4-b966-b50a5203b07a") : secret "isvc-primary-cfe5a5-predictor-serving-cert" not found
Jul 31 23:58:01.412338 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.412322    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kserve-provision-location\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.412486 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.412470    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.412524 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.412472    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-cabundle-cert\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.421929 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.421908    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jmsq\" (UniqueName: \"kubernetes.io/projected/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kube-api-access-8jmsq\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.459554 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.459526    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:01.915230 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.915199    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-proxy-tls\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:01.917551 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:01.917524    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-proxy-tls\") pod \"isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") " pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:02.135556 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:02.135518    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:02.263688 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:02.263666    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"]
Jul 31 23:58:02.266142 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:58:02.266118    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5fabb1_b92b_4ef4_b966_b50a5203b07a.slice/crio-ed70c89454c71e0d10c1aa3cdb6e7f5c21f4a4da476684c05457f8df2c3d0e54 WatchSource:0}: Error finding container ed70c89454c71e0d10c1aa3cdb6e7f5c21f4a4da476684c05457f8df2c3d0e54: Status 404 returned error can't find the container with id ed70c89454c71e0d10c1aa3cdb6e7f5c21f4a4da476684c05457f8df2c3d0e54
Jul 31 23:58:02.800622 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:02.800580    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 31 23:58:03.136574 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:03.136538    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" event={"ID":"ec5fabb1-b92b-4ef4-b966-b50a5203b07a","Type":"ContainerStarted","Data":"74d256382a1d8b2fdc81a8687c5beeeb17c11e94b3e82f2f9d4aff14e0427a64"}
Jul 31 23:58:03.136911 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:03.136622    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" event={"ID":"ec5fabb1-b92b-4ef4-b966-b50a5203b07a","Type":"ContainerStarted","Data":"ed70c89454c71e0d10c1aa3cdb6e7f5c21f4a4da476684c05457f8df2c3d0e54"}
Jul 31 23:58:04.143348 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:04.143317    2546 generic.go:356] "Generic (PLEG): container finished" podID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerID="413da9d06ab1b5f07eed9744287e3f231efe594118a34a193de689b20a105182" exitCode=0
Jul 31 23:58:04.143649 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:04.143389    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" event={"ID":"8584ee88-5809-4014-87d7-6cfdc124f7c0","Type":"ContainerDied","Data":"413da9d06ab1b5f07eed9744287e3f231efe594118a34a193de689b20a105182"}
Jul 31 23:58:05.148511 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:05.148480    2546 generic.go:356] "Generic (PLEG): container finished" podID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerID="ce077cd9f6d05adeaab196fe4c1beecbcb7ca6f84765399653ab46a106541d0a" exitCode=0
Jul 31 23:58:05.148896 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:05.148549    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" event={"ID":"f88fbfdf-09ad-408a-87a5-e674fc64b373","Type":"ContainerDied","Data":"ce077cd9f6d05adeaab196fe4c1beecbcb7ca6f84765399653ab46a106541d0a"}
Jul 31 23:58:06.154576 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:06.154538    2546 generic.go:356] "Generic (PLEG): container finished" podID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerID="74d256382a1d8b2fdc81a8687c5beeeb17c11e94b3e82f2f9d4aff14e0427a64" exitCode=0
Jul 31 23:58:06.154960 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:06.154614    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" event={"ID":"ec5fabb1-b92b-4ef4-b966-b50a5203b07a","Type":"ContainerDied","Data":"74d256382a1d8b2fdc81a8687c5beeeb17c11e94b3e82f2f9d4aff14e0427a64"}
Jul 31 23:58:06.157194 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:06.157112    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" event={"ID":"f88fbfdf-09ad-408a-87a5-e674fc64b373","Type":"ContainerStarted","Data":"cba87c39228107b699e015a497c524d8c9a2bace8013b94c1e1cb449fb0adb5b"}
Jul 31 23:58:06.157194 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:06.157164    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" event={"ID":"f88fbfdf-09ad-408a-87a5-e674fc64b373","Type":"ContainerStarted","Data":"f16439178a12e440da1eff882435a672554161300d9cde72d5256856b19fccb4"}
Jul 31 23:58:06.157194 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:06.157188    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" event={"ID":"f88fbfdf-09ad-408a-87a5-e674fc64b373","Type":"ContainerStarted","Data":"94544f6fb1500c3d3a6bae76d195bdf7b48b2a7b6fc8b34aec0d4d7d09e90f2a"}
Jul 31 23:58:06.157417 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:06.157398    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:06.157468 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:06.157459    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:06.159165 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:06.159143    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:5000: connect: connection refused"
Jul 31 23:58:06.190721 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:06.190673    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podStartSLOduration=2.555332276 podStartE2EDuration="6.190660597s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.635328321s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:58:06.189618423 +0000 UTC m=+716.649081235" watchObservedRunningTime="2026-07-31 23:58:06.190660597 +0000 UTC m=+716.650123410"
Jul 31 23:58:07.163487 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:07.163455    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" event={"ID":"ec5fabb1-b92b-4ef4-b966-b50a5203b07a","Type":"ContainerStarted","Data":"35575685ddcedc3c10b02edd0997238bca8bf8507f676d402aa1f0af724069c4"}
Jul 31 23:58:07.163906 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:07.163516    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" event={"ID":"ec5fabb1-b92b-4ef4-b966-b50a5203b07a","Type":"ContainerStarted","Data":"4ef05a4664b0811374cf0ed26886aeb147b587d69316bc5895544f30f94f8d91"}
Jul 31 23:58:07.163906 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:07.163682    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:07.163906 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:07.163866    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:07.163906 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:07.163893    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:5000: connect: connection refused"
Jul 31 23:58:07.165092 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:07.165062    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:58:07.185417 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:07.185380    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" podStartSLOduration=2.7398229069999998 podStartE2EDuration="6.185367218s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.445544311s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:58:07.183217476 +0000 UTC m=+717.642680289" watchObservedRunningTime="2026-07-31 23:58:07.185367218 +0000 UTC m=+717.644830030"
Jul 31 23:58:07.800962 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:07.800923    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 31 23:58:07.806273 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:07.806235    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:58:07.806653 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:07.806627    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:58:08.168399 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:08.168355    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:5000: connect: connection refused"
Jul 31 23:58:08.168800 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:08.168368    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:08.168869 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:08.168825    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:58:08.169793 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:08.169767    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 23:58:09.172951 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:09.172897    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 23:58:12.800208 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:12.800163    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 31 23:58:12.800702 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:12.800337    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:58:13.174217 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:13.174186    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:58:13.174684 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:13.174657    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:5000: connect: connection refused"
Jul 31 23:58:13.174998 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:13.174977    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:58:14.177601 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:14.177569    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:58:14.178153 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:14.178127    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 23:58:17.800537 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:17.800496    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 31 23:58:17.805855 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:17.805826    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:58:17.806128 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:17.806107    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:58:22.800157 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:22.800113    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 31 23:58:23.175348 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:23.175311    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:5000: connect: connection refused"
Jul 31 23:58:23.175730 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:23.175705    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:58:24.178269 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:24.178215    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 23:58:27.800544 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:27.800500    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 31 23:58:27.805938 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:27.805910    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 23:58:27.806070 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:27.806053    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:58:27.806306 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:27.806285    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:58:27.806398 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:27.806386    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:58:30.102872 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.102849    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:58:30.249660 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.249577    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvfj7\" (UniqueName: \"kubernetes.io/projected/8584ee88-5809-4014-87d7-6cfdc124f7c0-kube-api-access-kvfj7\") pod \"8584ee88-5809-4014-87d7-6cfdc124f7c0\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") "
Jul 31 23:58:30.249660 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.249637    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8584ee88-5809-4014-87d7-6cfdc124f7c0-cabundle-cert\") pod \"8584ee88-5809-4014-87d7-6cfdc124f7c0\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") "
Jul 31 23:58:30.249660 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.249658    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8584ee88-5809-4014-87d7-6cfdc124f7c0-kube-rbac-proxy-sar-config\") pod \"8584ee88-5809-4014-87d7-6cfdc124f7c0\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") "
Jul 31 23:58:30.249914 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.249685    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8584ee88-5809-4014-87d7-6cfdc124f7c0-kserve-provision-location\") pod \"8584ee88-5809-4014-87d7-6cfdc124f7c0\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") "
Jul 31 23:58:30.249914 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.249723    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8584ee88-5809-4014-87d7-6cfdc124f7c0-proxy-tls\") pod \"8584ee88-5809-4014-87d7-6cfdc124f7c0\" (UID: \"8584ee88-5809-4014-87d7-6cfdc124f7c0\") "
Jul 31 23:58:30.250027 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.250007    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8584ee88-5809-4014-87d7-6cfdc124f7c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8584ee88-5809-4014-87d7-6cfdc124f7c0" (UID: "8584ee88-5809-4014-87d7-6cfdc124f7c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 23:58:30.250087 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.250059    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8584ee88-5809-4014-87d7-6cfdc124f7c0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8584ee88-5809-4014-87d7-6cfdc124f7c0" (UID: "8584ee88-5809-4014-87d7-6cfdc124f7c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:58:30.250144 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.250090    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8584ee88-5809-4014-87d7-6cfdc124f7c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8584ee88-5809-4014-87d7-6cfdc124f7c0" (UID: "8584ee88-5809-4014-87d7-6cfdc124f7c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 23:58:30.251883 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.251851    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8584ee88-5809-4014-87d7-6cfdc124f7c0-kube-api-access-kvfj7" (OuterVolumeSpecName: "kube-api-access-kvfj7") pod "8584ee88-5809-4014-87d7-6cfdc124f7c0" (UID: "8584ee88-5809-4014-87d7-6cfdc124f7c0"). InnerVolumeSpecName "kube-api-access-kvfj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 23:58:30.251883 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.251870    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8584ee88-5809-4014-87d7-6cfdc124f7c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8584ee88-5809-4014-87d7-6cfdc124f7c0" (UID: "8584ee88-5809-4014-87d7-6cfdc124f7c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 23:58:30.280290 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.280265    2546 generic.go:356] "Generic (PLEG): container finished" podID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerID="2111257af11dd3bd5b0c71250d6ea010d85e1a8dcbc69043ad82ea846ea9c84b" exitCode=0
Jul 31 23:58:30.280403 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.280319    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"
Jul 31 23:58:30.280403 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.280340    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" event={"ID":"8584ee88-5809-4014-87d7-6cfdc124f7c0","Type":"ContainerDied","Data":"2111257af11dd3bd5b0c71250d6ea010d85e1a8dcbc69043ad82ea846ea9c84b"}
Jul 31 23:58:30.280403 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.280393    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69" event={"ID":"8584ee88-5809-4014-87d7-6cfdc124f7c0","Type":"ContainerDied","Data":"217ab0a92773deffd6eef105954c39d3450e437de19a6378e6665319d75cff1c"}
Jul 31 23:58:30.280527 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.280424    2546 scope.go:117] "RemoveContainer" containerID="2111257af11dd3bd5b0c71250d6ea010d85e1a8dcbc69043ad82ea846ea9c84b"
Jul 31 23:58:30.288335 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.288319    2546 scope.go:117] "RemoveContainer" containerID="a488a9fba1b861df9eed0ff8de0468a1464ddbff83c380111ea0d02229bf6c88"
Jul 31 23:58:30.296478 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.296460    2546 scope.go:117] "RemoveContainer" containerID="413da9d06ab1b5f07eed9744287e3f231efe594118a34a193de689b20a105182"
Jul 31 23:58:30.302240 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.302218    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"]
Jul 31 23:58:30.304018 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.304001    2546 scope.go:117] "RemoveContainer" containerID="3ee32e0e1f9928ab16f82754266edef28f338c98c374e0592613c1cf3ff02a75"
Jul 31 23:58:30.307715 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.307692    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7f9c4f966-f7v69"]
Jul 31 23:58:30.311075 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.311053    2546 scope.go:117] "RemoveContainer" containerID="2111257af11dd3bd5b0c71250d6ea010d85e1a8dcbc69043ad82ea846ea9c84b"
Jul 31 23:58:30.311361 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:58:30.311341    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2111257af11dd3bd5b0c71250d6ea010d85e1a8dcbc69043ad82ea846ea9c84b\": container with ID starting with 2111257af11dd3bd5b0c71250d6ea010d85e1a8dcbc69043ad82ea846ea9c84b not found: ID does not exist" containerID="2111257af11dd3bd5b0c71250d6ea010d85e1a8dcbc69043ad82ea846ea9c84b"
Jul 31 23:58:30.311444 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.311373    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2111257af11dd3bd5b0c71250d6ea010d85e1a8dcbc69043ad82ea846ea9c84b"} err="failed to get container status \"2111257af11dd3bd5b0c71250d6ea010d85e1a8dcbc69043ad82ea846ea9c84b\": rpc error: code = NotFound desc = could not find container \"2111257af11dd3bd5b0c71250d6ea010d85e1a8dcbc69043ad82ea846ea9c84b\": container with ID starting with 2111257af11dd3bd5b0c71250d6ea010d85e1a8dcbc69043ad82ea846ea9c84b not found: ID does not exist"
Jul 31 23:58:30.311444 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.311396    2546 scope.go:117] "RemoveContainer" containerID="a488a9fba1b861df9eed0ff8de0468a1464ddbff83c380111ea0d02229bf6c88"
Jul 31 23:58:30.311660 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:58:30.311640    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a488a9fba1b861df9eed0ff8de0468a1464ddbff83c380111ea0d02229bf6c88\": container with ID starting with a488a9fba1b861df9eed0ff8de0468a1464ddbff83c380111ea0d02229bf6c88 not found: ID does not exist" containerID="a488a9fba1b861df9eed0ff8de0468a1464ddbff83c380111ea0d02229bf6c88"
Jul 31 23:58:30.311702 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.311667    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a488a9fba1b861df9eed0ff8de0468a1464ddbff83c380111ea0d02229bf6c88"} err="failed to get container status \"a488a9fba1b861df9eed0ff8de0468a1464ddbff83c380111ea0d02229bf6c88\": rpc error: code = NotFound desc = could not find container \"a488a9fba1b861df9eed0ff8de0468a1464ddbff83c380111ea0d02229bf6c88\": container with ID starting with a488a9fba1b861df9eed0ff8de0468a1464ddbff83c380111ea0d02229bf6c88 not found: ID does not exist"
Jul 31 23:58:30.311702 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.311684    2546 scope.go:117] "RemoveContainer" containerID="413da9d06ab1b5f07eed9744287e3f231efe594118a34a193de689b20a105182"
Jul 31 23:58:30.311907 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:58:30.311889    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"413da9d06ab1b5f07eed9744287e3f231efe594118a34a193de689b20a105182\": container with ID starting with 413da9d06ab1b5f07eed9744287e3f231efe594118a34a193de689b20a105182 not found: ID does not exist" containerID="413da9d06ab1b5f07eed9744287e3f231efe594118a34a193de689b20a105182"
Jul 31 23:58:30.311953 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.311911    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413da9d06ab1b5f07eed9744287e3f231efe594118a34a193de689b20a105182"} err="failed to get container status \"413da9d06ab1b5f07eed9744287e3f231efe594118a34a193de689b20a105182\": rpc error: code = NotFound desc = could not find container \"413da9d06ab1b5f07eed9744287e3f231efe594118a34a193de689b20a105182\": container with ID starting with 413da9d06ab1b5f07eed9744287e3f231efe594118a34a193de689b20a105182 not found: ID does not exist"
Jul 31 23:58:30.311953 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.311926    2546 scope.go:117] "RemoveContainer" containerID="3ee32e0e1f9928ab16f82754266edef28f338c98c374e0592613c1cf3ff02a75"
Jul 31 23:58:30.312137 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:58:30.312119    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee32e0e1f9928ab16f82754266edef28f338c98c374e0592613c1cf3ff02a75\": container with ID starting with 3ee32e0e1f9928ab16f82754266edef28f338c98c374e0592613c1cf3ff02a75 not found: ID does not exist" containerID="3ee32e0e1f9928ab16f82754266edef28f338c98c374e0592613c1cf3ff02a75"
Jul 31 23:58:30.312216 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.312147    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee32e0e1f9928ab16f82754266edef28f338c98c374e0592613c1cf3ff02a75"} err="failed to get container status \"3ee32e0e1f9928ab16f82754266edef28f338c98c374e0592613c1cf3ff02a75\": rpc error: code = NotFound desc = could not find container \"3ee32e0e1f9928ab16f82754266edef28f338c98c374e0592613c1cf3ff02a75\": container with ID starting with 3ee32e0e1f9928ab16f82754266edef28f338c98c374e0592613c1cf3ff02a75 not found: ID does not exist"
Jul 31 23:58:30.351136 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.351117    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvfj7\" (UniqueName: \"kubernetes.io/projected/8584ee88-5809-4014-87d7-6cfdc124f7c0-kube-api-access-kvfj7\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:58:30.351216 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.351138    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8584ee88-5809-4014-87d7-6cfdc124f7c0-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:58:30.351216 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.351149    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8584ee88-5809-4014-87d7-6cfdc124f7c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:58:30.351216 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.351158    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8584ee88-5809-4014-87d7-6cfdc124f7c0-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:58:30.351216 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:30.351167    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8584ee88-5809-4014-87d7-6cfdc124f7c0-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Jul 31 23:58:32.081783 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:32.081750    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" path="/var/lib/kubelet/pods/8584ee88-5809-4014-87d7-6cfdc124f7c0/volumes"
Jul 31 23:58:33.174931 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:33.174891    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:5000: connect: connection refused"
Jul 31 23:58:33.175390 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:33.175364    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:58:34.177994 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:34.177955    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 23:58:40.464292 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:40.464244    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:43.174799 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:43.174752    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:5000: connect: connection refused"
Jul 31 23:58:43.175195 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:43.175119    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:58:44.178049 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:44.178005    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 23:58:51.995408 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:51.995364    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:58:53.174959 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:53.174922    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:5000: connect: connection refused"
Jul 31 23:58:53.175447 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:53.175351    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:58:54.178305 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:58:54.178248    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 23:59:03.175641 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:03.175594    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:5000: connect: connection refused"
Jul 31 23:59:03.176131 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:03.176105    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:59:04.178548 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:04.178509    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 23:59:13.174989 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:13.174949    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:5000: connect: connection refused"
Jul 31 23:59:13.175442 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:13.175378    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:59:14.178888 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:14.178837    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 23:59:23.175422 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:23.175389    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:59:23.175902 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:23.175688    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:59:24.179361 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:24.179330    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Jul 31 23:59:35.233916 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:35.233876    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:35.281213 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:35.281108    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"]
Jul 31 23:59:35.281493 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:35.281457    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container" containerID="cri-o://94544f6fb1500c3d3a6bae76d195bdf7b48b2a7b6fc8b34aec0d4d7d09e90f2a" gracePeriod=30
Jul 31 23:59:35.281636 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:35.281473    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent" containerID="cri-o://cba87c39228107b699e015a497c524d8c9a2bace8013b94c1e1cb449fb0adb5b" gracePeriod=30
Jul 31 23:59:35.281636 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:35.281544    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kube-rbac-proxy" containerID="cri-o://f16439178a12e440da1eff882435a672554161300d9cde72d5256856b19fccb4" gracePeriod=30
Jul 31 23:59:35.306299 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:35.306277    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:35.392348 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:35.392322    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:35.554122 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:35.554046    2546 generic.go:356] "Generic (PLEG): container finished" podID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerID="f16439178a12e440da1eff882435a672554161300d9cde72d5256856b19fccb4" exitCode=2
Jul 31 23:59:35.554248 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:35.554116    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" event={"ID":"f88fbfdf-09ad-408a-87a5-e674fc64b373","Type":"ContainerDied","Data":"f16439178a12e440da1eff882435a672554161300d9cde72d5256856b19fccb4"}
Jul 31 23:59:38.169131 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:38.169089    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 31 23:59:39.573690 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:39.573656    2546 generic.go:356] "Generic (PLEG): container finished" podID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerID="94544f6fb1500c3d3a6bae76d195bdf7b48b2a7b6fc8b34aec0d4d7d09e90f2a" exitCode=0
Jul 31 23:59:39.574034 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:39.573728    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" event={"ID":"f88fbfdf-09ad-408a-87a5-e674fc64b373","Type":"ContainerDied","Data":"94544f6fb1500c3d3a6bae76d195bdf7b48b2a7b6fc8b34aec0d4d7d09e90f2a"}
Jul 31 23:59:43.168860 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:43.168820    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 31 23:59:43.174948 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:43.174915    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:5000: connect: connection refused"
Jul 31 23:59:43.175265 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:43.175230    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:59:45.234949 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.234901    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:45.275153 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.275120    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"]
Jul 31 23:59:45.275514 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.275500    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="agent"
Jul 31 23:59:45.275562 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.275516    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="agent"
Jul 31 23:59:45.275562 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.275538    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="storage-initializer"
Jul 31 23:59:45.275562 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.275544    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="storage-initializer"
Jul 31 23:59:45.275562 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.275554    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kube-rbac-proxy"
Jul 31 23:59:45.275562 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.275560    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kube-rbac-proxy"
Jul 31 23:59:45.275719 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.275570    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kserve-container"
Jul 31 23:59:45.275719 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.275575    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kserve-container"
Jul 31 23:59:45.275719 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.275625    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kube-rbac-proxy"
Jul 31 23:59:45.275719 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.275632    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="kserve-container"
Jul 31 23:59:45.275719 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.275640    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8584ee88-5809-4014-87d7-6cfdc124f7c0" containerName="agent"
Jul 31 23:59:45.279122 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.279103    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:45.281596 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.281574    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 31 23:59:45.281687 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.281583    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 31 23:59:45.289384 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.289365    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"]
Jul 31 23:59:45.435554 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.435525    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-cabundle-cert\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:45.435724 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.435560    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kserve-provision-location\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:45.435724 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.435589    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:45.435724 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.435612    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-proxy-tls\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:45.435873 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.435727    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6t7\" (UniqueName: \"kubernetes.io/projected/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kube-api-access-tb6t7\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:45.536818 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.536738    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-cabundle-cert\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:45.536818 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.536782    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kserve-provision-location\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:45.536818 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.536809    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:45.537075 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.536835    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-proxy-tls\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:45.537075 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.536910    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6t7\" (UniqueName: \"kubernetes.io/projected/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kube-api-access-tb6t7\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:45.537075 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:59:45.537054    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 31 23:59:45.537218 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:59:45.537121    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-proxy-tls podName:9c3f67f0-5972-47c4-9bff-3a35eb5873f2 nodeName:}" failed. No retries permitted until 2026-07-31 23:59:46.037099906 +0000 UTC m=+816.496562700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-proxy-tls") pod "isvc-logger-predictor-6d6fd8ddb9-frctp" (UID: "9c3f67f0-5972-47c4-9bff-3a35eb5873f2") : secret "isvc-logger-predictor-serving-cert" not found
Jul 31 23:59:45.537319 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.537239    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kserve-provision-location\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:45.537457 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.537438    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-cabundle-cert\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:45.537497 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.537458    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:45.545589 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.545567    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6t7\" (UniqueName: \"kubernetes.io/projected/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kube-api-access-tb6t7\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:45.717660 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:45.717636    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:46.042549 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.042506    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-proxy-tls\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:46.044910 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.044888    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-proxy-tls\") pod \"isvc-logger-predictor-6d6fd8ddb9-frctp\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:46.189678 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.189647    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:46.326970 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.326898    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"]
Jul 31 23:59:46.329954 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:59:46.329926    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3f67f0_5972_47c4_9bff_3a35eb5873f2.slice/crio-62b47abe158d83b504079fe2e383831d2edf7c11f7d1f1ecc825e93e170f5ad3 WatchSource:0}: Error finding container 62b47abe158d83b504079fe2e383831d2edf7c11f7d1f1ecc825e93e170f5ad3: Status 404 returned error can't find the container with id 62b47abe158d83b504079fe2e383831d2edf7c11f7d1f1ecc825e93e170f5ad3
Jul 31 23:59:46.374046 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.374011    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:46.402874 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.402850    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"]
Jul 31 23:59:46.406356 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.406342    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:46.408910 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.408885    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-cfe5a5\""
Jul 31 23:59:46.408910 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.408902    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-cfe5a5-kube-rbac-proxy-sar-config\""
Jul 31 23:59:46.409077 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.408925    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-cfe5a5-dockercfg-7hxqs\""
Jul 31 23:59:46.409077 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.408960    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-cfe5a5-predictor-serving-cert\""
Jul 31 23:59:46.416506 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.416486    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"]
Jul 31 23:59:46.548563 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.548527    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/523268d8-96b9-42ad-8730-42a7b318b103-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:46.548726 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.548573    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbckv\" (UniqueName: \"kubernetes.io/projected/523268d8-96b9-42ad-8730-42a7b318b103-kube-api-access-nbckv\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:46.548726 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.548595    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/523268d8-96b9-42ad-8730-42a7b318b103-kserve-provision-location\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:46.548823 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.548748    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/523268d8-96b9-42ad-8730-42a7b318b103-proxy-tls\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:46.548823 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.548789    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/523268d8-96b9-42ad-8730-42a7b318b103-cabundle-cert\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:46.605459 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.605391    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" event={"ID":"9c3f67f0-5972-47c4-9bff-3a35eb5873f2","Type":"ContainerStarted","Data":"e000a28ec3bcbf8fdc4bb198c1e0aa90d03a051b68c141d84b883ee9f86cb114"}
Jul 31 23:59:46.605594 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.605477    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" event={"ID":"9c3f67f0-5972-47c4-9bff-3a35eb5873f2","Type":"ContainerStarted","Data":"62b47abe158d83b504079fe2e383831d2edf7c11f7d1f1ecc825e93e170f5ad3"}
Jul 31 23:59:46.650095 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.650065    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/523268d8-96b9-42ad-8730-42a7b318b103-proxy-tls\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:46.650281 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.650106    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/523268d8-96b9-42ad-8730-42a7b318b103-cabundle-cert\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:46.650281 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.650187    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/523268d8-96b9-42ad-8730-42a7b318b103-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:46.650281 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:59:46.650205    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-serving-cert: secret "isvc-secondary-cfe5a5-predictor-serving-cert" not found
Jul 31 23:59:46.650281 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.650226    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbckv\" (UniqueName: \"kubernetes.io/projected/523268d8-96b9-42ad-8730-42a7b318b103-kube-api-access-nbckv\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:46.650281 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.650277    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/523268d8-96b9-42ad-8730-42a7b318b103-kserve-provision-location\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:46.650559 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:59:46.650281    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/523268d8-96b9-42ad-8730-42a7b318b103-proxy-tls podName:523268d8-96b9-42ad-8730-42a7b318b103 nodeName:}" failed. No retries permitted until 2026-07-31 23:59:47.150247157 +0000 UTC m=+817.609709948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/523268d8-96b9-42ad-8730-42a7b318b103-proxy-tls") pod "isvc-secondary-cfe5a5-predictor-756547584d-zpm5g" (UID: "523268d8-96b9-42ad-8730-42a7b318b103") : secret "isvc-secondary-cfe5a5-predictor-serving-cert" not found
Jul 31 23:59:46.650650 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.650625    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/523268d8-96b9-42ad-8730-42a7b318b103-kserve-provision-location\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:46.650824 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.650808    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/523268d8-96b9-42ad-8730-42a7b318b103-cabundle-cert\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:46.650883 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.650865    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/523268d8-96b9-42ad-8730-42a7b318b103-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:46.659084 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.659065    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbckv\" (UniqueName: \"kubernetes.io/projected/523268d8-96b9-42ad-8730-42a7b318b103-kube-api-access-nbckv\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:46.916769 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:46.916738    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Jul 31 23:59:47.153871 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:47.153838    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/523268d8-96b9-42ad-8730-42a7b318b103-proxy-tls\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:47.156235 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:47.156205    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/523268d8-96b9-42ad-8730-42a7b318b103-proxy-tls\") pod \"isvc-secondary-cfe5a5-predictor-756547584d-zpm5g\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") " pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:47.316636 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:47.316550    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Jul 31 23:59:47.441941 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:47.441916    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"]
Jul 31 23:59:47.444186 ip-10-0-140-138 kubenswrapper[2546]: W0731 23:59:47.444160    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523268d8_96b9_42ad_8730_42a7b318b103.slice/crio-90d697dd3bc26c3b907bc118103786b42fbcdee49c3a633f8b2f43768388dbfd WatchSource:0}: Error finding container 90d697dd3bc26c3b907bc118103786b42fbcdee49c3a633f8b2f43768388dbfd: Status 404 returned error can't find the container with id 90d697dd3bc26c3b907bc118103786b42fbcdee49c3a633f8b2f43768388dbfd
Jul 31 23:59:47.612100 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:47.612060    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g" event={"ID":"523268d8-96b9-42ad-8730-42a7b318b103","Type":"ContainerStarted","Data":"dfce334c2b06c6c41c5c6f6123a07df414ed824c474309007c243bb31b6a7b90"}
Jul 31 23:59:47.612292 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:47.612135    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g" event={"ID":"523268d8-96b9-42ad-8730-42a7b318b103","Type":"ContainerStarted","Data":"90d697dd3bc26c3b907bc118103786b42fbcdee49c3a633f8b2f43768388dbfd"}
Jul 31 23:59:48.168950 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:48.168910    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 31 23:59:48.169183 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:48.169035    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Jul 31 23:59:50.627262 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:50.627216    2546 generic.go:356] "Generic (PLEG): container finished" podID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerID="e000a28ec3bcbf8fdc4bb198c1e0aa90d03a051b68c141d84b883ee9f86cb114" exitCode=0
Jul 31 23:59:50.627700 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:50.627289    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" event={"ID":"9c3f67f0-5972-47c4-9bff-3a35eb5873f2","Type":"ContainerDied","Data":"e000a28ec3bcbf8fdc4bb198c1e0aa90d03a051b68c141d84b883ee9f86cb114"}
Jul 31 23:59:50.628975 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:50.628956    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cfe5a5-predictor-756547584d-zpm5g_523268d8-96b9-42ad-8730-42a7b318b103/storage-initializer/0.log"
Jul 31 23:59:50.629084 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:50.628991    2546 generic.go:356] "Generic (PLEG): container finished" podID="523268d8-96b9-42ad-8730-42a7b318b103" containerID="dfce334c2b06c6c41c5c6f6123a07df414ed824c474309007c243bb31b6a7b90" exitCode=1
Jul 31 23:59:50.629084 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:50.629040    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g" event={"ID":"523268d8-96b9-42ad-8730-42a7b318b103","Type":"ContainerDied","Data":"dfce334c2b06c6c41c5c6f6123a07df414ed824c474309007c243bb31b6a7b90"}
Jul 31 23:59:51.635532 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:51.635503    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cfe5a5-predictor-756547584d-zpm5g_523268d8-96b9-42ad-8730-42a7b318b103/storage-initializer/0.log"
Jul 31 23:59:51.635944 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:51.635620    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g" event={"ID":"523268d8-96b9-42ad-8730-42a7b318b103","Type":"ContainerStarted","Data":"8b2328128517fc0cacbf622866a8d5c3266f4ceccd1cf6cb6adb9f56a4042e9b"}
Jul 31 23:59:51.637959 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:51.637937    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" event={"ID":"9c3f67f0-5972-47c4-9bff-3a35eb5873f2","Type":"ContainerStarted","Data":"be92b96b4005a5a9f6f86bae011c445c7af4b58754d588a8f088c6c9b0f3c5ff"}
Jul 31 23:59:51.638057 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:51.637992    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" event={"ID":"9c3f67f0-5972-47c4-9bff-3a35eb5873f2","Type":"ContainerStarted","Data":"76c7daebb7d992d3f3bda3ac831e9ba8ced5b0163cddc8957fc2d0ac1d528f95"}
Jul 31 23:59:51.638057 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:51.638015    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" event={"ID":"9c3f67f0-5972-47c4-9bff-3a35eb5873f2","Type":"ContainerStarted","Data":"249a30f80f48b55bbb82783b1538d9f32dd5e1ff09460412e22b5367c0ccccda"}
Jul 31 23:59:51.638303 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:51.638283    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:51.638563 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:51.638333    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:51.638563 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:51.638358    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:51.640161 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:51.640129    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 31 23:59:51.640842 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:51.640818    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:59:51.675493 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:51.675445    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podStartSLOduration=3.123463537 podStartE2EDuration="6.67542896s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.551965423s" isStatefulPod=false podCreationTimestamp="2026-07-31 23:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 23:59:51.674809574 +0000 UTC m=+822.134272400" watchObservedRunningTime="2026-07-31 23:59:51.67542896 +0000 UTC m=+822.134891770"
Jul 31 23:59:52.642307 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:52.642269    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 31 23:59:52.642674 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:52.642634    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:59:53.168479 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:53.168440    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 31 23:59:53.174917 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:53.174883    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:5000: connect: connection refused"
Jul 31 23:59:53.175298 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:53.175273    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:59:57.651905 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:57.651880    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Jul 31 23:59:57.652395 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:57.652364    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 31 23:59:57.652730 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:57.652708    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 23:59:57.667533 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:57.667514    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cfe5a5-predictor-756547584d-zpm5g_523268d8-96b9-42ad-8730-42a7b318b103/storage-initializer/1.log"
Jul 31 23:59:57.667869 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:57.667856    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cfe5a5-predictor-756547584d-zpm5g_523268d8-96b9-42ad-8730-42a7b318b103/storage-initializer/0.log"
Jul 31 23:59:57.667935 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:57.667884    2546 generic.go:356] "Generic (PLEG): container finished" podID="523268d8-96b9-42ad-8730-42a7b318b103" containerID="8b2328128517fc0cacbf622866a8d5c3266f4ceccd1cf6cb6adb9f56a4042e9b" exitCode=1
Jul 31 23:59:57.667989 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:57.667971    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g" event={"ID":"523268d8-96b9-42ad-8730-42a7b318b103","Type":"ContainerDied","Data":"8b2328128517fc0cacbf622866a8d5c3266f4ceccd1cf6cb6adb9f56a4042e9b"}
Jul 31 23:59:57.668057 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:57.668042    2546 scope.go:117] "RemoveContainer" containerID="dfce334c2b06c6c41c5c6f6123a07df414ed824c474309007c243bb31b6a7b90"
Jul 31 23:59:57.668423 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:57.668406    2546 scope.go:117] "RemoveContainer" containerID="dfce334c2b06c6c41c5c6f6123a07df414ed824c474309007c243bb31b6a7b90"
Jul 31 23:59:57.679064 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:59:57.679041    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-cfe5a5-predictor-756547584d-zpm5g_kserve-ci-e2e-test(523268d8-96b9-42ad-8730-42a7b318b103)\"" pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g" podUID="523268d8-96b9-42ad-8730-42a7b318b103"
Jul 31 23:59:57.679567 ip-10-0-140-138 kubenswrapper[2546]: E0731 23:59:57.679542    2546 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-cfe5a5-predictor-756547584d-zpm5g_kserve-ci-e2e-test_523268d8-96b9-42ad-8730-42a7b318b103_0 in pod sandbox 90d697dd3bc26c3b907bc118103786b42fbcdee49c3a633f8b2f43768388dbfd from index: no such id: 'dfce334c2b06c6c41c5c6f6123a07df414ed824c474309007c243bb31b6a7b90'" containerID="dfce334c2b06c6c41c5c6f6123a07df414ed824c474309007c243bb31b6a7b90"
Jul 31 23:59:57.679651 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:57.679577    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfce334c2b06c6c41c5c6f6123a07df414ed824c474309007c243bb31b6a7b90"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-cfe5a5-predictor-756547584d-zpm5g_kserve-ci-e2e-test_523268d8-96b9-42ad-8730-42a7b318b103_0 in pod sandbox 90d697dd3bc26c3b907bc118103786b42fbcdee49c3a633f8b2f43768388dbfd from index: no such id: 'dfce334c2b06c6c41c5c6f6123a07df414ed824c474309007c243bb31b6a7b90'"
Jul 31 23:59:58.168837 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:58.168794    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 31 23:59:58.674373 ip-10-0-140-138 kubenswrapper[2546]: I0731 23:59:58.674349    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cfe5a5-predictor-756547584d-zpm5g_523268d8-96b9-42ad-8730-42a7b318b103/storage-initializer/1.log"
Aug 01 00:00:02.444613 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.444580    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"]
Aug 01 00:00:02.464892 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.464855    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:02.486985 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.486959    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"]
Aug 01 00:00:02.487213 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.487172    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kserve-container" containerID="cri-o://4ef05a4664b0811374cf0ed26886aeb147b587d69316bc5895544f30f94f8d91" gracePeriod=30
Aug 01 00:00:02.487443 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.487418    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kube-rbac-proxy" containerID="cri-o://35575685ddcedc3c10b02edd0997238bca8bf8507f676d402aa1f0af724069c4" gracePeriod=30
Aug 01 00:00:02.520791 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.520720    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:02.607609 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.607587    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cfe5a5-predictor-756547584d-zpm5g_523268d8-96b9-42ad-8730-42a7b318b103/storage-initializer/1.log"
Aug 01 00:00:02.609872 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.609854    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Aug 01 00:00:02.680035 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.680012    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/523268d8-96b9-42ad-8730-42a7b318b103-proxy-tls\") pod \"523268d8-96b9-42ad-8730-42a7b318b103\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") "
Aug 01 00:00:02.680158 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.680074    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbckv\" (UniqueName: \"kubernetes.io/projected/523268d8-96b9-42ad-8730-42a7b318b103-kube-api-access-nbckv\") pod \"523268d8-96b9-42ad-8730-42a7b318b103\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") "
Aug 01 00:00:02.680158 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.680132    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/523268d8-96b9-42ad-8730-42a7b318b103-kserve-provision-location\") pod \"523268d8-96b9-42ad-8730-42a7b318b103\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") "
Aug 01 00:00:02.680294 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.680163    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/523268d8-96b9-42ad-8730-42a7b318b103-kube-rbac-proxy-sar-config\") pod \"523268d8-96b9-42ad-8730-42a7b318b103\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") "
Aug 01 00:00:02.680294 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.680214    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/523268d8-96b9-42ad-8730-42a7b318b103-cabundle-cert\") pod \"523268d8-96b9-42ad-8730-42a7b318b103\" (UID: \"523268d8-96b9-42ad-8730-42a7b318b103\") "
Aug 01 00:00:02.680530 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.680481    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523268d8-96b9-42ad-8730-42a7b318b103-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "523268d8-96b9-42ad-8730-42a7b318b103" (UID: "523268d8-96b9-42ad-8730-42a7b318b103"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:02.680633 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.680551    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/523268d8-96b9-42ad-8730-42a7b318b103-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "523268d8-96b9-42ad-8730-42a7b318b103" (UID: "523268d8-96b9-42ad-8730-42a7b318b103"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:02.680633 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.680614    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/523268d8-96b9-42ad-8730-42a7b318b103-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "523268d8-96b9-42ad-8730-42a7b318b103" (UID: "523268d8-96b9-42ad-8730-42a7b318b103"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:02.682053 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.682034    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/523268d8-96b9-42ad-8730-42a7b318b103-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "523268d8-96b9-42ad-8730-42a7b318b103" (UID: "523268d8-96b9-42ad-8730-42a7b318b103"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:00:02.682477 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.682461    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523268d8-96b9-42ad-8730-42a7b318b103-kube-api-access-nbckv" (OuterVolumeSpecName: "kube-api-access-nbckv") pod "523268d8-96b9-42ad-8730-42a7b318b103" (UID: "523268d8-96b9-42ad-8730-42a7b318b103"). InnerVolumeSpecName "kube-api-access-nbckv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:00:02.692329 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.692312    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cfe5a5-predictor-756547584d-zpm5g_523268d8-96b9-42ad-8730-42a7b318b103/storage-initializer/1.log"
Aug 01 00:00:02.692423 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.692398    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"
Aug 01 00:00:02.692491 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.692423    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g" event={"ID":"523268d8-96b9-42ad-8730-42a7b318b103","Type":"ContainerDied","Data":"90d697dd3bc26c3b907bc118103786b42fbcdee49c3a633f8b2f43768388dbfd"}
Aug 01 00:00:02.692548 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.692496    2546 scope.go:117] "RemoveContainer" containerID="8b2328128517fc0cacbf622866a8d5c3266f4ceccd1cf6cb6adb9f56a4042e9b"
Aug 01 00:00:02.694965 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.694913    2546 generic.go:356] "Generic (PLEG): container finished" podID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerID="35575685ddcedc3c10b02edd0997238bca8bf8507f676d402aa1f0af724069c4" exitCode=2
Aug 01 00:00:02.694965 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.694952    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" event={"ID":"ec5fabb1-b92b-4ef4-b966-b50a5203b07a","Type":"ContainerDied","Data":"35575685ddcedc3c10b02edd0997238bca8bf8507f676d402aa1f0af724069c4"}
Aug 01 00:00:02.733672 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.733645    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"]
Aug 01 00:00:02.736827 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.736804    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cfe5a5-predictor-756547584d-zpm5g"]
Aug 01 00:00:02.781673 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.781649    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/523268d8-96b9-42ad-8730-42a7b318b103-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:02.781673 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.781672    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/523268d8-96b9-42ad-8730-42a7b318b103-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:02.781806 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.781683    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/523268d8-96b9-42ad-8730-42a7b318b103-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:02.781806 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.781696    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/523268d8-96b9-42ad-8730-42a7b318b103-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:02.781806 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:02.781707    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbckv\" (UniqueName: \"kubernetes.io/projected/523268d8-96b9-42ad-8730-42a7b318b103-kube-api-access-nbckv\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:03.168494 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:03.168456    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Aug 01 00:00:03.175186 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:03.175149    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:5000: connect: connection refused"
Aug 01 00:00:03.175358 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:03.175341    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Aug 01 00:00:03.175551 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:03.175525    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:00:03.175632 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:03.175622    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Aug 01 00:00:04.076967 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:04.076936    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="523268d8-96b9-42ad-8730-42a7b318b103" path="/var/lib/kubelet/pods/523268d8-96b9-42ad-8730-42a7b318b103/volumes"
Aug 01 00:00:04.174007 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:04.173973    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Aug 01 00:00:04.178380 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:04.178356    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Aug 01 00:00:05.426429 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.426400    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Aug 01 00:00:05.506119 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.506093    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88fbfdf-09ad-408a-87a5-e674fc64b373-proxy-tls\") pod \"f88fbfdf-09ad-408a-87a5-e674fc64b373\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") "
Aug 01 00:00:05.506319 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.506148    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f88fbfdf-09ad-408a-87a5-e674fc64b373-cabundle-cert\") pod \"f88fbfdf-09ad-408a-87a5-e674fc64b373\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") "
Aug 01 00:00:05.506319 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.506179    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f88fbfdf-09ad-408a-87a5-e674fc64b373-kube-rbac-proxy-sar-config\") pod \"f88fbfdf-09ad-408a-87a5-e674fc64b373\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") "
Aug 01 00:00:05.506319 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.506228    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f88fbfdf-09ad-408a-87a5-e674fc64b373-kserve-provision-location\") pod \"f88fbfdf-09ad-408a-87a5-e674fc64b373\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") "
Aug 01 00:00:05.506319 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.506316    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwtcb\" (UniqueName: \"kubernetes.io/projected/f88fbfdf-09ad-408a-87a5-e674fc64b373-kube-api-access-dwtcb\") pod \"f88fbfdf-09ad-408a-87a5-e674fc64b373\" (UID: \"f88fbfdf-09ad-408a-87a5-e674fc64b373\") "
Aug 01 00:00:05.506569 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.506540    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88fbfdf-09ad-408a-87a5-e674fc64b373-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f88fbfdf-09ad-408a-87a5-e674fc64b373" (UID: "f88fbfdf-09ad-408a-87a5-e674fc64b373"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:05.506612 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.506561    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88fbfdf-09ad-408a-87a5-e674fc64b373-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f88fbfdf-09ad-408a-87a5-e674fc64b373" (UID: "f88fbfdf-09ad-408a-87a5-e674fc64b373"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:05.506612 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.506553    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88fbfdf-09ad-408a-87a5-e674fc64b373-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f88fbfdf-09ad-408a-87a5-e674fc64b373" (UID: "f88fbfdf-09ad-408a-87a5-e674fc64b373"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:05.508188 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.508166    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f88fbfdf-09ad-408a-87a5-e674fc64b373-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f88fbfdf-09ad-408a-87a5-e674fc64b373" (UID: "f88fbfdf-09ad-408a-87a5-e674fc64b373"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:00:05.508292 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.508269    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88fbfdf-09ad-408a-87a5-e674fc64b373-kube-api-access-dwtcb" (OuterVolumeSpecName: "kube-api-access-dwtcb") pod "f88fbfdf-09ad-408a-87a5-e674fc64b373" (UID: "f88fbfdf-09ad-408a-87a5-e674fc64b373"). InnerVolumeSpecName "kube-api-access-dwtcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:00:05.606936 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.606867    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwtcb\" (UniqueName: \"kubernetes.io/projected/f88fbfdf-09ad-408a-87a5-e674fc64b373-kube-api-access-dwtcb\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:05.606936 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.606894    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f88fbfdf-09ad-408a-87a5-e674fc64b373-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:05.606936 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.606905    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f88fbfdf-09ad-408a-87a5-e674fc64b373-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:05.606936 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.606912    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f88fbfdf-09ad-408a-87a5-e674fc64b373-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:05.606936 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.606921    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f88fbfdf-09ad-408a-87a5-e674fc64b373-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:05.712781 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.712750    2546 generic.go:356] "Generic (PLEG): container finished" podID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerID="cba87c39228107b699e015a497c524d8c9a2bace8013b94c1e1cb449fb0adb5b" exitCode=0
Aug 01 00:00:05.712906 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.712812    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"
Aug 01 00:00:05.712906 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.712838    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" event={"ID":"f88fbfdf-09ad-408a-87a5-e674fc64b373","Type":"ContainerDied","Data":"cba87c39228107b699e015a497c524d8c9a2bace8013b94c1e1cb449fb0adb5b"}
Aug 01 00:00:05.712996 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.712915    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj" event={"ID":"f88fbfdf-09ad-408a-87a5-e674fc64b373","Type":"ContainerDied","Data":"6a6cb47ee8864f191f77979132b2430490db8d621741b4a149c6c5f23e656845"}
Aug 01 00:00:05.712996 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.712971    2546 scope.go:117] "RemoveContainer" containerID="cba87c39228107b699e015a497c524d8c9a2bace8013b94c1e1cb449fb0adb5b"
Aug 01 00:00:05.721992 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.721975    2546 scope.go:117] "RemoveContainer" containerID="f16439178a12e440da1eff882435a672554161300d9cde72d5256856b19fccb4"
Aug 01 00:00:05.729216 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.729201    2546 scope.go:117] "RemoveContainer" containerID="94544f6fb1500c3d3a6bae76d195bdf7b48b2a7b6fc8b34aec0d4d7d09e90f2a"
Aug 01 00:00:05.734573 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.734552    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"]
Aug 01 00:00:05.736594 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.736578    2546 scope.go:117] "RemoveContainer" containerID="ce077cd9f6d05adeaab196fe4c1beecbcb7ca6f84765399653ab46a106541d0a"
Aug 01 00:00:05.740446 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.740428    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6b67ff98b9-phfhj"]
Aug 01 00:00:05.744158 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.744143    2546 scope.go:117] "RemoveContainer" containerID="cba87c39228107b699e015a497c524d8c9a2bace8013b94c1e1cb449fb0adb5b"
Aug 01 00:00:05.744424 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:00:05.744399    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba87c39228107b699e015a497c524d8c9a2bace8013b94c1e1cb449fb0adb5b\": container with ID starting with cba87c39228107b699e015a497c524d8c9a2bace8013b94c1e1cb449fb0adb5b not found: ID does not exist" containerID="cba87c39228107b699e015a497c524d8c9a2bace8013b94c1e1cb449fb0adb5b"
Aug 01 00:00:05.744469 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.744432    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba87c39228107b699e015a497c524d8c9a2bace8013b94c1e1cb449fb0adb5b"} err="failed to get container status \"cba87c39228107b699e015a497c524d8c9a2bace8013b94c1e1cb449fb0adb5b\": rpc error: code = NotFound desc = could not find container \"cba87c39228107b699e015a497c524d8c9a2bace8013b94c1e1cb449fb0adb5b\": container with ID starting with cba87c39228107b699e015a497c524d8c9a2bace8013b94c1e1cb449fb0adb5b not found: ID does not exist"
Aug 01 00:00:05.744469 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.744450    2546 scope.go:117] "RemoveContainer" containerID="f16439178a12e440da1eff882435a672554161300d9cde72d5256856b19fccb4"
Aug 01 00:00:05.744633 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:00:05.744611    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f16439178a12e440da1eff882435a672554161300d9cde72d5256856b19fccb4\": container with ID starting with f16439178a12e440da1eff882435a672554161300d9cde72d5256856b19fccb4 not found: ID does not exist" containerID="f16439178a12e440da1eff882435a672554161300d9cde72d5256856b19fccb4"
Aug 01 00:00:05.744673 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.744641    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f16439178a12e440da1eff882435a672554161300d9cde72d5256856b19fccb4"} err="failed to get container status \"f16439178a12e440da1eff882435a672554161300d9cde72d5256856b19fccb4\": rpc error: code = NotFound desc = could not find container \"f16439178a12e440da1eff882435a672554161300d9cde72d5256856b19fccb4\": container with ID starting with f16439178a12e440da1eff882435a672554161300d9cde72d5256856b19fccb4 not found: ID does not exist"
Aug 01 00:00:05.744673 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.744655    2546 scope.go:117] "RemoveContainer" containerID="94544f6fb1500c3d3a6bae76d195bdf7b48b2a7b6fc8b34aec0d4d7d09e90f2a"
Aug 01 00:00:05.744874 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:00:05.744860    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94544f6fb1500c3d3a6bae76d195bdf7b48b2a7b6fc8b34aec0d4d7d09e90f2a\": container with ID starting with 94544f6fb1500c3d3a6bae76d195bdf7b48b2a7b6fc8b34aec0d4d7d09e90f2a not found: ID does not exist" containerID="94544f6fb1500c3d3a6bae76d195bdf7b48b2a7b6fc8b34aec0d4d7d09e90f2a"
Aug 01 00:00:05.744909 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.744876    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94544f6fb1500c3d3a6bae76d195bdf7b48b2a7b6fc8b34aec0d4d7d09e90f2a"} err="failed to get container status \"94544f6fb1500c3d3a6bae76d195bdf7b48b2a7b6fc8b34aec0d4d7d09e90f2a\": rpc error: code = NotFound desc = could not find container \"94544f6fb1500c3d3a6bae76d195bdf7b48b2a7b6fc8b34aec0d4d7d09e90f2a\": container with ID starting with 94544f6fb1500c3d3a6bae76d195bdf7b48b2a7b6fc8b34aec0d4d7d09e90f2a not found: ID does not exist"
Aug 01 00:00:05.744909 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.744888    2546 scope.go:117] "RemoveContainer" containerID="ce077cd9f6d05adeaab196fe4c1beecbcb7ca6f84765399653ab46a106541d0a"
Aug 01 00:00:05.745094 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:00:05.745079    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce077cd9f6d05adeaab196fe4c1beecbcb7ca6f84765399653ab46a106541d0a\": container with ID starting with ce077cd9f6d05adeaab196fe4c1beecbcb7ca6f84765399653ab46a106541d0a not found: ID does not exist" containerID="ce077cd9f6d05adeaab196fe4c1beecbcb7ca6f84765399653ab46a106541d0a"
Aug 01 00:00:05.745135 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:05.745097    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce077cd9f6d05adeaab196fe4c1beecbcb7ca6f84765399653ab46a106541d0a"} err="failed to get container status \"ce077cd9f6d05adeaab196fe4c1beecbcb7ca6f84765399653ab46a106541d0a\": rpc error: code = NotFound desc = could not find container \"ce077cd9f6d05adeaab196fe4c1beecbcb7ca6f84765399653ab46a106541d0a\": container with ID starting with ce077cd9f6d05adeaab196fe4c1beecbcb7ca6f84765399653ab46a106541d0a not found: ID does not exist"
Aug 01 00:00:06.075938 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:06.075903    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" path="/var/lib/kubelet/pods/f88fbfdf-09ad-408a-87a5-e674fc64b373/volumes"
Aug 01 00:00:06.721273 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:06.721224    2546 generic.go:356] "Generic (PLEG): container finished" podID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerID="4ef05a4664b0811374cf0ed26886aeb147b587d69316bc5895544f30f94f8d91" exitCode=0
Aug 01 00:00:06.721581 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:06.721284    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" event={"ID":"ec5fabb1-b92b-4ef4-b966-b50a5203b07a","Type":"ContainerDied","Data":"4ef05a4664b0811374cf0ed26886aeb147b587d69316bc5895544f30f94f8d91"}
Aug 01 00:00:06.832429 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:06.832409    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Aug 01 00:00:06.918739 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:06.918663    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kube-rbac-proxy-sar-config\") pod \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") "
Aug 01 00:00:06.918739 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:06.918701    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kserve-provision-location\") pod \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") "
Aug 01 00:00:06.918958 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:06.918743    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jmsq\" (UniqueName: \"kubernetes.io/projected/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kube-api-access-8jmsq\") pod \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") "
Aug 01 00:00:06.918958 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:06.918768    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-cabundle-cert\") pod \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") "
Aug 01 00:00:06.918958 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:06.918813    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-proxy-tls\") pod \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\" (UID: \"ec5fabb1-b92b-4ef4-b966-b50a5203b07a\") "
Aug 01 00:00:06.919108 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:06.919046    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec5fabb1-b92b-4ef4-b966-b50a5203b07a" (UID: "ec5fabb1-b92b-4ef4-b966-b50a5203b07a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:06.919108 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:06.919057    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec5fabb1-b92b-4ef4-b966-b50a5203b07a" (UID: "ec5fabb1-b92b-4ef4-b966-b50a5203b07a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:06.919218 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:06.919163    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ec5fabb1-b92b-4ef4-b966-b50a5203b07a" (UID: "ec5fabb1-b92b-4ef4-b966-b50a5203b07a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:06.920886 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:06.920857    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kube-api-access-8jmsq" (OuterVolumeSpecName: "kube-api-access-8jmsq") pod "ec5fabb1-b92b-4ef4-b966-b50a5203b07a" (UID: "ec5fabb1-b92b-4ef4-b966-b50a5203b07a"). InnerVolumeSpecName "kube-api-access-8jmsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:00:06.920886 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:06.920859    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec5fabb1-b92b-4ef4-b966-b50a5203b07a" (UID: "ec5fabb1-b92b-4ef4-b966-b50a5203b07a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:00:07.020429 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.020401    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:07.020429 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.020425    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jmsq\" (UniqueName: \"kubernetes.io/projected/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kube-api-access-8jmsq\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:07.020575 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.020436    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:07.020575 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.020446    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:07.020575 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.020454    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec5fabb1-b92b-4ef4-b966-b50a5203b07a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:07.653153 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.653110    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Aug 01 00:00:07.653644 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.653622    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:00:07.662024 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.662000    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:07.692271 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692206    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"]
Aug 01 00:00:07.692580 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692565    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kube-rbac-proxy"
Aug 01 00:00:07.692619 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692581    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kube-rbac-proxy"
Aug 01 00:00:07.692619 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692592    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kube-rbac-proxy"
Aug 01 00:00:07.692619 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692597    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kube-rbac-proxy"
Aug 01 00:00:07.692619 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692605    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="storage-initializer"
Aug 01 00:00:07.692619 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692611    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="storage-initializer"
Aug 01 00:00:07.692619 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692620    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kserve-container"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692626    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kserve-container"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692640    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692646    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692652    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692657    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692663    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="523268d8-96b9-42ad-8730-42a7b318b103" containerName="storage-initializer"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692668    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="523268d8-96b9-42ad-8730-42a7b318b103" containerName="storage-initializer"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692673    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="523268d8-96b9-42ad-8730-42a7b318b103" containerName="storage-initializer"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692678    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="523268d8-96b9-42ad-8730-42a7b318b103" containerName="storage-initializer"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692686    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="storage-initializer"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692691    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="storage-initializer"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692738    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="523268d8-96b9-42ad-8730-42a7b318b103" containerName="storage-initializer"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692744    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kube-rbac-proxy"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692751    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="523268d8-96b9-42ad-8730-42a7b318b103" containerName="storage-initializer"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692758    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="kserve-container"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692764    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f88fbfdf-09ad-408a-87a5-e674fc64b373" containerName="agent"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692770    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kube-rbac-proxy"
Aug 01 00:00:07.692796 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.692775    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" containerName="kserve-container"
Aug 01 00:00:07.695877 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.695864    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:07.699098 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.699076    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-719348-predictor-serving-cert\""
Aug 01 00:00:07.699098 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.699093    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-719348-kube-rbac-proxy-sar-config\""
Aug 01 00:00:07.699283 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.699079    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-719348\""
Aug 01 00:00:07.699283 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.699125    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-719348-dockercfg-qsnfh\""
Aug 01 00:00:07.706399 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.706379    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"]
Aug 01 00:00:07.730006 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.729835    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c40a5be-bcdd-4393-8114-254fccee6068-kserve-provision-location\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:07.730006 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.729893    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c40a5be-bcdd-4393-8114-254fccee6068-proxy-tls\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:07.730006 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.729926    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c40a5be-bcdd-4393-8114-254fccee6068-cabundle-cert\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:07.730543 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.730124    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5mt6\" (UniqueName: \"kubernetes.io/projected/7c40a5be-bcdd-4393-8114-254fccee6068-kube-api-access-r5mt6\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:07.730543 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.730218    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c40a5be-bcdd-4393-8114-254fccee6068-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:07.734996 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.734973    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm" event={"ID":"ec5fabb1-b92b-4ef4-b966-b50a5203b07a","Type":"ContainerDied","Data":"ed70c89454c71e0d10c1aa3cdb6e7f5c21f4a4da476684c05457f8df2c3d0e54"}
Aug 01 00:00:07.735109 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.734995    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"
Aug 01 00:00:07.735109 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.735049    2546 scope.go:117] "RemoveContainer" containerID="35575685ddcedc3c10b02edd0997238bca8bf8507f676d402aa1f0af724069c4"
Aug 01 00:00:07.743044 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.742981    2546 scope.go:117] "RemoveContainer" containerID="4ef05a4664b0811374cf0ed26886aeb147b587d69316bc5895544f30f94f8d91"
Aug 01 00:00:07.750136 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.750122    2546 scope.go:117] "RemoveContainer" containerID="74d256382a1d8b2fdc81a8687c5beeeb17c11e94b3e82f2f9d4aff14e0427a64"
Aug 01 00:00:07.757274 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.757216    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"]
Aug 01 00:00:07.766754 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.766736    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cfe5a5-predictor-68cc5778dc-2cdsm"]
Aug 01 00:00:07.830984 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.830959    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c40a5be-bcdd-4393-8114-254fccee6068-proxy-tls\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:07.831104 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.830993    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c40a5be-bcdd-4393-8114-254fccee6068-cabundle-cert\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:07.831104 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.831041    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5mt6\" (UniqueName: \"kubernetes.io/projected/7c40a5be-bcdd-4393-8114-254fccee6068-kube-api-access-r5mt6\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:07.831104 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.831095    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c40a5be-bcdd-4393-8114-254fccee6068-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:07.831280 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:00:07.831131    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-719348-predictor-serving-cert: secret "isvc-init-fail-719348-predictor-serving-cert" not found
Aug 01 00:00:07.831280 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.831154    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c40a5be-bcdd-4393-8114-254fccee6068-kserve-provision-location\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:07.831280 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:00:07.831196    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c40a5be-bcdd-4393-8114-254fccee6068-proxy-tls podName:7c40a5be-bcdd-4393-8114-254fccee6068 nodeName:}" failed. No retries permitted until 2026-08-01 00:00:08.331175338 +0000 UTC m=+838.790638138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7c40a5be-bcdd-4393-8114-254fccee6068-proxy-tls") pod "isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s" (UID: "7c40a5be-bcdd-4393-8114-254fccee6068") : secret "isvc-init-fail-719348-predictor-serving-cert" not found
Aug 01 00:00:07.831528 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.831508    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c40a5be-bcdd-4393-8114-254fccee6068-kserve-provision-location\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:07.831764 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.831742    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c40a5be-bcdd-4393-8114-254fccee6068-cabundle-cert\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:07.831825 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.831766    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c40a5be-bcdd-4393-8114-254fccee6068-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:07.839961 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:07.839939    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5mt6\" (UniqueName: \"kubernetes.io/projected/7c40a5be-bcdd-4393-8114-254fccee6068-kube-api-access-r5mt6\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:08.075962 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:08.075887    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec5fabb1-b92b-4ef4-b966-b50a5203b07a" path="/var/lib/kubelet/pods/ec5fabb1-b92b-4ef4-b966-b50a5203b07a/volumes"
Aug 01 00:00:08.202584 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:08.202551    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:08.335232 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:08.335160    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c40a5be-bcdd-4393-8114-254fccee6068-proxy-tls\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:08.337287 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:08.337271    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c40a5be-bcdd-4393-8114-254fccee6068-proxy-tls\") pod \"isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") " pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:08.606219 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:08.606128    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:08.736501 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:08.736468    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"]
Aug 01 00:00:08.739286 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:00:08.739240    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c40a5be_bcdd_4393_8114_254fccee6068.slice/crio-c6a91a2b86eb82775450ecc9ca1856bbc19d0df5033446427045f79d35a27a9e WatchSource:0}: Error finding container c6a91a2b86eb82775450ecc9ca1856bbc19d0df5033446427045f79d35a27a9e: Status 404 returned error can't find the container with id c6a91a2b86eb82775450ecc9ca1856bbc19d0df5033446427045f79d35a27a9e
Aug 01 00:00:09.746089 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:09.746055    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s" event={"ID":"7c40a5be-bcdd-4393-8114-254fccee6068","Type":"ContainerStarted","Data":"35b464f4925f73280b813f9a1482b2a59d93fb9d1c1b1066909e21e9e77b20f2"}
Aug 01 00:00:09.746459 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:09.746111    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s" event={"ID":"7c40a5be-bcdd-4393-8114-254fccee6068","Type":"ContainerStarted","Data":"c6a91a2b86eb82775450ecc9ca1856bbc19d0df5033446427045f79d35a27a9e"}
Aug 01 00:00:15.770698 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:15.770630    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s_7c40a5be-bcdd-4393-8114-254fccee6068/storage-initializer/0.log"
Aug 01 00:00:15.770698 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:15.770666    2546 generic.go:356] "Generic (PLEG): container finished" podID="7c40a5be-bcdd-4393-8114-254fccee6068" containerID="35b464f4925f73280b813f9a1482b2a59d93fb9d1c1b1066909e21e9e77b20f2" exitCode=1
Aug 01 00:00:15.771073 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:15.770721    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s" event={"ID":"7c40a5be-bcdd-4393-8114-254fccee6068","Type":"ContainerDied","Data":"35b464f4925f73280b813f9a1482b2a59d93fb9d1c1b1066909e21e9e77b20f2"}
Aug 01 00:00:16.776961 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:16.776929    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s_7c40a5be-bcdd-4393-8114-254fccee6068/storage-initializer/0.log"
Aug 01 00:00:16.777357 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:16.777046    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s" event={"ID":"7c40a5be-bcdd-4393-8114-254fccee6068","Type":"ContainerStarted","Data":"24579a1681256437593fb90342c40ce0434e7aa26e81bbdbfad4086155564d1c"}
Aug 01 00:00:17.652909 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:17.652861    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Aug 01 00:00:17.653357 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:17.653333    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:00:17.693695 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:17.693663    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"]
Aug 01 00:00:17.724999 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:17.724964    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:17.781212 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:17.781172    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s" podUID="7c40a5be-bcdd-4393-8114-254fccee6068" containerName="storage-initializer" containerID="cri-o://24579a1681256437593fb90342c40ce0434e7aa26e81bbdbfad4086155564d1c" gracePeriod=30
Aug 01 00:00:20.432591 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.432569    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s_7c40a5be-bcdd-4393-8114-254fccee6068/storage-initializer/1.log"
Aug 01 00:00:20.433035 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.433017    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s_7c40a5be-bcdd-4393-8114-254fccee6068/storage-initializer/0.log"
Aug 01 00:00:20.435417 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.435400    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:20.534794 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.534737    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c40a5be-bcdd-4393-8114-254fccee6068-proxy-tls\") pod \"7c40a5be-bcdd-4393-8114-254fccee6068\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") "
Aug 01 00:00:20.534794 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.534769    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c40a5be-bcdd-4393-8114-254fccee6068-kube-rbac-proxy-sar-config\") pod \"7c40a5be-bcdd-4393-8114-254fccee6068\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") "
Aug 01 00:00:20.534945 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.534830    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5mt6\" (UniqueName: \"kubernetes.io/projected/7c40a5be-bcdd-4393-8114-254fccee6068-kube-api-access-r5mt6\") pod \"7c40a5be-bcdd-4393-8114-254fccee6068\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") "
Aug 01 00:00:20.534945 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.534852    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c40a5be-bcdd-4393-8114-254fccee6068-cabundle-cert\") pod \"7c40a5be-bcdd-4393-8114-254fccee6068\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") "
Aug 01 00:00:20.534945 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.534878    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c40a5be-bcdd-4393-8114-254fccee6068-kserve-provision-location\") pod \"7c40a5be-bcdd-4393-8114-254fccee6068\" (UID: \"7c40a5be-bcdd-4393-8114-254fccee6068\") "
Aug 01 00:00:20.535288 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.535239    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c40a5be-bcdd-4393-8114-254fccee6068-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c40a5be-bcdd-4393-8114-254fccee6068" (UID: "7c40a5be-bcdd-4393-8114-254fccee6068"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:00:20.535288 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.535247    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c40a5be-bcdd-4393-8114-254fccee6068-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c40a5be-bcdd-4393-8114-254fccee6068" (UID: "7c40a5be-bcdd-4393-8114-254fccee6068"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:20.535460 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.535247    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c40a5be-bcdd-4393-8114-254fccee6068-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7c40a5be-bcdd-4393-8114-254fccee6068" (UID: "7c40a5be-bcdd-4393-8114-254fccee6068"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:00:20.536749 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.536731    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c40a5be-bcdd-4393-8114-254fccee6068-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c40a5be-bcdd-4393-8114-254fccee6068" (UID: "7c40a5be-bcdd-4393-8114-254fccee6068"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:00:20.536818 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.536784    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c40a5be-bcdd-4393-8114-254fccee6068-kube-api-access-r5mt6" (OuterVolumeSpecName: "kube-api-access-r5mt6") pod "7c40a5be-bcdd-4393-8114-254fccee6068" (UID: "7c40a5be-bcdd-4393-8114-254fccee6068"). InnerVolumeSpecName "kube-api-access-r5mt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:00:20.635559 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.635536    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c40a5be-bcdd-4393-8114-254fccee6068-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:20.635559 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.635556    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c40a5be-bcdd-4393-8114-254fccee6068-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:20.635691 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.635566    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5mt6\" (UniqueName: \"kubernetes.io/projected/7c40a5be-bcdd-4393-8114-254fccee6068-kube-api-access-r5mt6\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:20.635691 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.635574    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c40a5be-bcdd-4393-8114-254fccee6068-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:20.635691 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.635582    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c40a5be-bcdd-4393-8114-254fccee6068-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:00:20.793694 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.793632    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s_7c40a5be-bcdd-4393-8114-254fccee6068/storage-initializer/1.log"
Aug 01 00:00:20.794033 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.794018    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s_7c40a5be-bcdd-4393-8114-254fccee6068/storage-initializer/0.log"
Aug 01 00:00:20.794103 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.794053    2546 generic.go:356] "Generic (PLEG): container finished" podID="7c40a5be-bcdd-4393-8114-254fccee6068" containerID="24579a1681256437593fb90342c40ce0434e7aa26e81bbdbfad4086155564d1c" exitCode=1
Aug 01 00:00:20.794142 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.794115    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"
Aug 01 00:00:20.794173 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.794144    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s" event={"ID":"7c40a5be-bcdd-4393-8114-254fccee6068","Type":"ContainerDied","Data":"24579a1681256437593fb90342c40ce0434e7aa26e81bbdbfad4086155564d1c"}
Aug 01 00:00:20.794220 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.794206    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s" event={"ID":"7c40a5be-bcdd-4393-8114-254fccee6068","Type":"ContainerDied","Data":"c6a91a2b86eb82775450ecc9ca1856bbc19d0df5033446427045f79d35a27a9e"}
Aug 01 00:00:20.794282 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.794240    2546 scope.go:117] "RemoveContainer" containerID="24579a1681256437593fb90342c40ce0434e7aa26e81bbdbfad4086155564d1c"
Aug 01 00:00:20.805051 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.804776    2546 scope.go:117] "RemoveContainer" containerID="35b464f4925f73280b813f9a1482b2a59d93fb9d1c1b1066909e21e9e77b20f2"
Aug 01 00:00:20.812552 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.812534    2546 scope.go:117] "RemoveContainer" containerID="24579a1681256437593fb90342c40ce0434e7aa26e81bbdbfad4086155564d1c"
Aug 01 00:00:20.812831 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:00:20.812812    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24579a1681256437593fb90342c40ce0434e7aa26e81bbdbfad4086155564d1c\": container with ID starting with 24579a1681256437593fb90342c40ce0434e7aa26e81bbdbfad4086155564d1c not found: ID does not exist" containerID="24579a1681256437593fb90342c40ce0434e7aa26e81bbdbfad4086155564d1c"
Aug 01 00:00:20.812890 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.812848    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24579a1681256437593fb90342c40ce0434e7aa26e81bbdbfad4086155564d1c"} err="failed to get container status \"24579a1681256437593fb90342c40ce0434e7aa26e81bbdbfad4086155564d1c\": rpc error: code = NotFound desc = could not find container \"24579a1681256437593fb90342c40ce0434e7aa26e81bbdbfad4086155564d1c\": container with ID starting with 24579a1681256437593fb90342c40ce0434e7aa26e81bbdbfad4086155564d1c not found: ID does not exist"
Aug 01 00:00:20.812890 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.812865    2546 scope.go:117] "RemoveContainer" containerID="35b464f4925f73280b813f9a1482b2a59d93fb9d1c1b1066909e21e9e77b20f2"
Aug 01 00:00:20.813121 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:00:20.813100    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b464f4925f73280b813f9a1482b2a59d93fb9d1c1b1066909e21e9e77b20f2\": container with ID starting with 35b464f4925f73280b813f9a1482b2a59d93fb9d1c1b1066909e21e9e77b20f2 not found: ID does not exist" containerID="35b464f4925f73280b813f9a1482b2a59d93fb9d1c1b1066909e21e9e77b20f2"
Aug 01 00:00:20.813173 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.813127    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b464f4925f73280b813f9a1482b2a59d93fb9d1c1b1066909e21e9e77b20f2"} err="failed to get container status \"35b464f4925f73280b813f9a1482b2a59d93fb9d1c1b1066909e21e9e77b20f2\": rpc error: code = NotFound desc = could not find container \"35b464f4925f73280b813f9a1482b2a59d93fb9d1c1b1066909e21e9e77b20f2\": container with ID starting with 35b464f4925f73280b813f9a1482b2a59d93fb9d1c1b1066909e21e9e77b20f2 not found: ID does not exist"
Aug 01 00:00:20.829951 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.829928    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"]
Aug 01 00:00:20.835458 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:20.835437    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-719348-predictor-7cbc5fc6db-djw4s"]
Aug 01 00:00:22.075594 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:22.075565    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c40a5be-bcdd-4393-8114-254fccee6068" path="/var/lib/kubelet/pods/7c40a5be-bcdd-4393-8114-254fccee6068/volumes"
Aug 01 00:00:22.903373 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:22.903342    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:23.180501 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:23.180422    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:00:27.653122 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:27.653074    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Aug 01 00:00:27.653564 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:27.653500    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:00:37.653300 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:37.653226    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Aug 01 00:00:37.653727 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:37.653702    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:00:47.653162 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:47.653119    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Aug 01 00:00:47.653649 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:47.653625    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:00:57.652908 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:57.652854    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Aug 01 00:00:57.653422 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:00:57.653302    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:01:07.653484 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:07.653398    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Aug 01 00:01:07.654078 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:07.654055    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Aug 01 00:01:20.494626 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:20.494589    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"]
Aug 01 00:01:20.497184 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:20.494907    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container" containerID="cri-o://249a30f80f48b55bbb82783b1538d9f32dd5e1ff09460412e22b5367c0ccccda" gracePeriod=30
Aug 01 00:01:20.497184 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:20.494999    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kube-rbac-proxy" containerID="cri-o://76c7daebb7d992d3f3bda3ac831e9ba8ced5b0163cddc8957fc2d0ac1d528f95" gracePeriod=30
Aug 01 00:01:20.497184 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:20.494980    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent" containerID="cri-o://be92b96b4005a5a9f6f86bae011c445c7af4b58754d588a8f088c6c9b0f3c5ff" gracePeriod=30
Aug 01 00:01:20.531510 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:20.531457    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:20.606179 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:20.606142    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:21.042994 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:21.042963    2546 generic.go:356] "Generic (PLEG): container finished" podID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerID="76c7daebb7d992d3f3bda3ac831e9ba8ced5b0163cddc8957fc2d0ac1d528f95" exitCode=2
Aug 01 00:01:21.043164 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:21.043039    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" event={"ID":"9c3f67f0-5972-47c4-9bff-3a35eb5873f2","Type":"ContainerDied","Data":"76c7daebb7d992d3f3bda3ac831e9ba8ced5b0163cddc8957fc2d0ac1d528f95"}
Aug 01 00:01:21.237807 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:21.237764    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:22.643142 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:22.643102    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Aug 01 00:01:24.294420 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:24.294384    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:24.394347 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:24.394311    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:01:26.063489 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:26.063453    2546 generic.go:356] "Generic (PLEG): container finished" podID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerID="249a30f80f48b55bbb82783b1538d9f32dd5e1ff09460412e22b5367c0ccccda" exitCode=0
Aug 01 00:01:26.063870 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:26.063520    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" event={"ID":"9c3f67f0-5972-47c4-9bff-3a35eb5873f2","Type":"ContainerDied","Data":"249a30f80f48b55bbb82783b1538d9f32dd5e1ff09460412e22b5367c0ccccda"}
Aug 01 00:01:27.643339 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:27.643294    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Aug 01 00:01:27.652908 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:27.652883    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Aug 01 00:01:27.654551 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:27.654521    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:01:32.643304 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:32.643242    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Aug 01 00:01:32.643775 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:32.643379    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Aug 01 00:01:37.643052 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:37.643010    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Aug 01 00:01:37.652879 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:37.652842    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Aug 01 00:01:37.654473 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:37.654446    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:01:42.642328 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:42.642288    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Aug 01 00:01:47.643086 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:47.643036    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Aug 01 00:01:47.652529 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:47.652500    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Aug 01 00:01:47.652629 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:47.652617    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Aug 01 00:01:47.655208 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:47.655183    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 01 00:01:47.655332 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:47.655319    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Aug 01 00:01:50.649092 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.649070    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Aug 01 00:01:50.675112 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.675087    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-cabundle-cert\") pod \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") "
Aug 01 00:01:50.675284 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.675121    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kserve-provision-location\") pod \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") "
Aug 01 00:01:50.675284 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.675138    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-proxy-tls\") pod \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") "
Aug 01 00:01:50.675284 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.675181    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kube-rbac-proxy-sar-config\") pod \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") "
Aug 01 00:01:50.675284 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.675232    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb6t7\" (UniqueName: \"kubernetes.io/projected/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kube-api-access-tb6t7\") pod \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\" (UID: \"9c3f67f0-5972-47c4-9bff-3a35eb5873f2\") "
Aug 01 00:01:50.675530 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.675496    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c3f67f0-5972-47c4-9bff-3a35eb5873f2" (UID: "9c3f67f0-5972-47c4-9bff-3a35eb5873f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:01:50.675587 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.675533    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9c3f67f0-5972-47c4-9bff-3a35eb5873f2" (UID: "9c3f67f0-5972-47c4-9bff-3a35eb5873f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:01:50.675587 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.675568    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9c3f67f0-5972-47c4-9bff-3a35eb5873f2" (UID: "9c3f67f0-5972-47c4-9bff-3a35eb5873f2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:01:50.677248 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.677223    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kube-api-access-tb6t7" (OuterVolumeSpecName: "kube-api-access-tb6t7") pod "9c3f67f0-5972-47c4-9bff-3a35eb5873f2" (UID: "9c3f67f0-5972-47c4-9bff-3a35eb5873f2"). InnerVolumeSpecName "kube-api-access-tb6t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:01:50.677371 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.677295    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9c3f67f0-5972-47c4-9bff-3a35eb5873f2" (UID: "9c3f67f0-5972-47c4-9bff-3a35eb5873f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:01:50.776614 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.776535    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:01:50.776614 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.776563    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb6t7\" (UniqueName: \"kubernetes.io/projected/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kube-api-access-tb6t7\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:01:50.776614 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.776572    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:01:50.776614 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.776581    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:01:50.776614 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:50.776590    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c3f67f0-5972-47c4-9bff-3a35eb5873f2-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:01:51.168168 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.168132    2546 generic.go:356] "Generic (PLEG): container finished" podID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerID="be92b96b4005a5a9f6f86bae011c445c7af4b58754d588a8f088c6c9b0f3c5ff" exitCode=0
Aug 01 00:01:51.168351 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.168206    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"
Aug 01 00:01:51.168351 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.168214    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" event={"ID":"9c3f67f0-5972-47c4-9bff-3a35eb5873f2","Type":"ContainerDied","Data":"be92b96b4005a5a9f6f86bae011c445c7af4b58754d588a8f088c6c9b0f3c5ff"}
Aug 01 00:01:51.168351 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.168295    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp" event={"ID":"9c3f67f0-5972-47c4-9bff-3a35eb5873f2","Type":"ContainerDied","Data":"62b47abe158d83b504079fe2e383831d2edf7c11f7d1f1ecc825e93e170f5ad3"}
Aug 01 00:01:51.168351 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.168328    2546 scope.go:117] "RemoveContainer" containerID="be92b96b4005a5a9f6f86bae011c445c7af4b58754d588a8f088c6c9b0f3c5ff"
Aug 01 00:01:51.176737 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.176718    2546 scope.go:117] "RemoveContainer" containerID="76c7daebb7d992d3f3bda3ac831e9ba8ced5b0163cddc8957fc2d0ac1d528f95"
Aug 01 00:01:51.184527 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.184508    2546 scope.go:117] "RemoveContainer" containerID="249a30f80f48b55bbb82783b1538d9f32dd5e1ff09460412e22b5367c0ccccda"
Aug 01 00:01:51.190131 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.190104    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"]
Aug 01 00:01:51.192495 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.192470    2546 scope.go:117] "RemoveContainer" containerID="e000a28ec3bcbf8fdc4bb198c1e0aa90d03a051b68c141d84b883ee9f86cb114"
Aug 01 00:01:51.194149 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.194128    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d6fd8ddb9-frctp"]
Aug 01 00:01:51.200183 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.200157    2546 scope.go:117] "RemoveContainer" containerID="be92b96b4005a5a9f6f86bae011c445c7af4b58754d588a8f088c6c9b0f3c5ff"
Aug 01 00:01:51.200447 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:01:51.200429    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be92b96b4005a5a9f6f86bae011c445c7af4b58754d588a8f088c6c9b0f3c5ff\": container with ID starting with be92b96b4005a5a9f6f86bae011c445c7af4b58754d588a8f088c6c9b0f3c5ff not found: ID does not exist" containerID="be92b96b4005a5a9f6f86bae011c445c7af4b58754d588a8f088c6c9b0f3c5ff"
Aug 01 00:01:51.200510 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.200457    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be92b96b4005a5a9f6f86bae011c445c7af4b58754d588a8f088c6c9b0f3c5ff"} err="failed to get container status \"be92b96b4005a5a9f6f86bae011c445c7af4b58754d588a8f088c6c9b0f3c5ff\": rpc error: code = NotFound desc = could not find container \"be92b96b4005a5a9f6f86bae011c445c7af4b58754d588a8f088c6c9b0f3c5ff\": container with ID starting with be92b96b4005a5a9f6f86bae011c445c7af4b58754d588a8f088c6c9b0f3c5ff not found: ID does not exist"
Aug 01 00:01:51.200510 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.200477    2546 scope.go:117] "RemoveContainer" containerID="76c7daebb7d992d3f3bda3ac831e9ba8ced5b0163cddc8957fc2d0ac1d528f95"
Aug 01 00:01:51.200676 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:01:51.200662    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c7daebb7d992d3f3bda3ac831e9ba8ced5b0163cddc8957fc2d0ac1d528f95\": container with ID starting with 76c7daebb7d992d3f3bda3ac831e9ba8ced5b0163cddc8957fc2d0ac1d528f95 not found: ID does not exist" containerID="76c7daebb7d992d3f3bda3ac831e9ba8ced5b0163cddc8957fc2d0ac1d528f95"
Aug 01 00:01:51.200721 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.200680    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c7daebb7d992d3f3bda3ac831e9ba8ced5b0163cddc8957fc2d0ac1d528f95"} err="failed to get container status \"76c7daebb7d992d3f3bda3ac831e9ba8ced5b0163cddc8957fc2d0ac1d528f95\": rpc error: code = NotFound desc = could not find container \"76c7daebb7d992d3f3bda3ac831e9ba8ced5b0163cddc8957fc2d0ac1d528f95\": container with ID starting with 76c7daebb7d992d3f3bda3ac831e9ba8ced5b0163cddc8957fc2d0ac1d528f95 not found: ID does not exist"
Aug 01 00:01:51.200721 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.200695    2546 scope.go:117] "RemoveContainer" containerID="249a30f80f48b55bbb82783b1538d9f32dd5e1ff09460412e22b5367c0ccccda"
Aug 01 00:01:51.200905 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:01:51.200886    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249a30f80f48b55bbb82783b1538d9f32dd5e1ff09460412e22b5367c0ccccda\": container with ID starting with 249a30f80f48b55bbb82783b1538d9f32dd5e1ff09460412e22b5367c0ccccda not found: ID does not exist" containerID="249a30f80f48b55bbb82783b1538d9f32dd5e1ff09460412e22b5367c0ccccda"
Aug 01 00:01:51.200983 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.200914    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249a30f80f48b55bbb82783b1538d9f32dd5e1ff09460412e22b5367c0ccccda"} err="failed to get container status \"249a30f80f48b55bbb82783b1538d9f32dd5e1ff09460412e22b5367c0ccccda\": rpc error: code = NotFound desc = could not find container \"249a30f80f48b55bbb82783b1538d9f32dd5e1ff09460412e22b5367c0ccccda\": container with ID starting with 249a30f80f48b55bbb82783b1538d9f32dd5e1ff09460412e22b5367c0ccccda not found: ID does not exist"
Aug 01 00:01:51.200983 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.200935    2546 scope.go:117] "RemoveContainer" containerID="e000a28ec3bcbf8fdc4bb198c1e0aa90d03a051b68c141d84b883ee9f86cb114"
Aug 01 00:01:51.201200 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:01:51.201181    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e000a28ec3bcbf8fdc4bb198c1e0aa90d03a051b68c141d84b883ee9f86cb114\": container with ID starting with e000a28ec3bcbf8fdc4bb198c1e0aa90d03a051b68c141d84b883ee9f86cb114 not found: ID does not exist" containerID="e000a28ec3bcbf8fdc4bb198c1e0aa90d03a051b68c141d84b883ee9f86cb114"
Aug 01 00:01:51.201243 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:51.201205    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e000a28ec3bcbf8fdc4bb198c1e0aa90d03a051b68c141d84b883ee9f86cb114"} err="failed to get container status \"e000a28ec3bcbf8fdc4bb198c1e0aa90d03a051b68c141d84b883ee9f86cb114\": rpc error: code = NotFound desc = could not find container \"e000a28ec3bcbf8fdc4bb198c1e0aa90d03a051b68c141d84b883ee9f86cb114\": container with ID starting with e000a28ec3bcbf8fdc4bb198c1e0aa90d03a051b68c141d84b883ee9f86cb114 not found: ID does not exist"
Aug 01 00:01:52.075239 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:01:52.075196    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" path="/var/lib/kubelet/pods/9c3f67f0-5972-47c4-9bff-3a35eb5873f2/volumes"
Aug 01 00:02:13.126284 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:13.126234    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:18.246450 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.246416    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:18.285373 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285337    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"]
Aug 01 00:02:18.285666 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285654    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="storage-initializer"
Aug 01 00:02:18.285712 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285667    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="storage-initializer"
Aug 01 00:02:18.285712 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285681    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container"
Aug 01 00:02:18.285712 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285687    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container"
Aug 01 00:02:18.285712 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285694    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c40a5be-bcdd-4393-8114-254fccee6068" containerName="storage-initializer"
Aug 01 00:02:18.285712 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285699    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c40a5be-bcdd-4393-8114-254fccee6068" containerName="storage-initializer"
Aug 01 00:02:18.285712 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285706    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kube-rbac-proxy"
Aug 01 00:02:18.285712 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285711    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kube-rbac-proxy"
Aug 01 00:02:18.285923 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285718    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent"
Aug 01 00:02:18.285923 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285724    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent"
Aug 01 00:02:18.285923 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285779    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kserve-container"
Aug 01 00:02:18.285923 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285787    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c40a5be-bcdd-4393-8114-254fccee6068" containerName="storage-initializer"
Aug 01 00:02:18.285923 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285794    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="agent"
Aug 01 00:02:18.285923 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285799    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c40a5be-bcdd-4393-8114-254fccee6068" containerName="storage-initializer"
Aug 01 00:02:18.285923 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285805    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c3f67f0-5972-47c4-9bff-3a35eb5873f2" containerName="kube-rbac-proxy"
Aug 01 00:02:18.285923 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285858    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c40a5be-bcdd-4393-8114-254fccee6068" containerName="storage-initializer"
Aug 01 00:02:18.285923 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.285863    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c40a5be-bcdd-4393-8114-254fccee6068" containerName="storage-initializer"
Aug 01 00:02:18.288751 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.288736    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.291328 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.291300    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Aug 01 00:02:18.291328 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.291319    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:02:18.291520 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.291335    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:02:18.291520 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.291319    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Aug 01 00:02:18.291630 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.291525    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:02:18.291630 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.291540    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:02:18.291731 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.291720    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:02:18.291787 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.291758    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"]
Aug 01 00:02:18.390537 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.390513    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23036ad5-7ce0-4f38-965d-f401a118cd10-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.390702 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.390557    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23036ad5-7ce0-4f38-965d-f401a118cd10-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.390702 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.390585    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23036ad5-7ce0-4f38-965d-f401a118cd10-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.390702 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.390641    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6ssq\" (UniqueName: \"kubernetes.io/projected/23036ad5-7ce0-4f38-965d-f401a118cd10-kube-api-access-q6ssq\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.390850 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.390683    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23036ad5-7ce0-4f38-965d-f401a118cd10-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.491571 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.491539    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23036ad5-7ce0-4f38-965d-f401a118cd10-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.491723 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.491587    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23036ad5-7ce0-4f38-965d-f401a118cd10-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.491723 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.491611    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23036ad5-7ce0-4f38-965d-f401a118cd10-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.491723 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.491638    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6ssq\" (UniqueName: \"kubernetes.io/projected/23036ad5-7ce0-4f38-965d-f401a118cd10-kube-api-access-q6ssq\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.491723 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.491659    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23036ad5-7ce0-4f38-965d-f401a118cd10-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.491723 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:02:18.491709    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Aug 01 00:02:18.491978 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:02:18.491792    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23036ad5-7ce0-4f38-965d-f401a118cd10-proxy-tls podName:23036ad5-7ce0-4f38-965d-f401a118cd10 nodeName:}" failed. No retries permitted until 2026-08-01 00:02:18.991770432 +0000 UTC m=+969.451233229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/23036ad5-7ce0-4f38-965d-f401a118cd10-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-bnvcw" (UID: "23036ad5-7ce0-4f38-965d-f401a118cd10") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Aug 01 00:02:18.491978 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.491952    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23036ad5-7ce0-4f38-965d-f401a118cd10-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.492246 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.492231    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23036ad5-7ce0-4f38-965d-f401a118cd10-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.492316 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.492304    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23036ad5-7ce0-4f38-965d-f401a118cd10-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.503272 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.503205    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6ssq\" (UniqueName: \"kubernetes.io/projected/23036ad5-7ce0-4f38-965d-f401a118cd10-kube-api-access-q6ssq\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.565900 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.565855    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:02:18.994717 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.994685    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23036ad5-7ce0-4f38-965d-f401a118cd10-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:18.996904 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:18.996876    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23036ad5-7ce0-4f38-965d-f401a118cd10-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bnvcw\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:19.200121 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:19.200094    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:19.330076 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:19.330053    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"]
Aug 01 00:02:19.332031 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:02:19.331999    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23036ad5_7ce0_4f38_965d_f401a118cd10.slice/crio-70943cb860ccb2f1136e09e8af382106866880635eb69aad71d8b6f8e7b7547a WatchSource:0}: Error finding container 70943cb860ccb2f1136e09e8af382106866880635eb69aad71d8b6f8e7b7547a: Status 404 returned error can't find the container with id 70943cb860ccb2f1136e09e8af382106866880635eb69aad71d8b6f8e7b7547a
Aug 01 00:02:20.279754 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:20.279715    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" event={"ID":"23036ad5-7ce0-4f38-965d-f401a118cd10","Type":"ContainerStarted","Data":"1e299b99ffe0097b52b4c121aff04ff8e13b309ae60572e63deef2a0f4ceecdb"}
Aug 01 00:02:20.279924 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:20.279783    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" event={"ID":"23036ad5-7ce0-4f38-965d-f401a118cd10","Type":"ContainerStarted","Data":"70943cb860ccb2f1136e09e8af382106866880635eb69aad71d8b6f8e7b7547a"}
Aug 01 00:02:23.292882 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:23.292850    2546 generic.go:356] "Generic (PLEG): container finished" podID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerID="1e299b99ffe0097b52b4c121aff04ff8e13b309ae60572e63deef2a0f4ceecdb" exitCode=0
Aug 01 00:02:23.292882 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:23.292885    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" event={"ID":"23036ad5-7ce0-4f38-965d-f401a118cd10","Type":"ContainerDied","Data":"1e299b99ffe0097b52b4c121aff04ff8e13b309ae60572e63deef2a0f4ceecdb"}
Aug 01 00:02:23.294105 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:23.294089    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:02:43.395628 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:43.395594    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" event={"ID":"23036ad5-7ce0-4f38-965d-f401a118cd10","Type":"ContainerStarted","Data":"d6668082cdb4354b432001a74a53b2ea506e5bc230cde5cbbff1e11dc6f41cdd"}
Aug 01 00:02:43.396024 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:43.395651    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" event={"ID":"23036ad5-7ce0-4f38-965d-f401a118cd10","Type":"ContainerStarted","Data":"0530081e195ebc9d6034c3ba7ff2ae949927721912cca5eeb85adc560396e635"}
Aug 01 00:02:43.396072 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:43.396053    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:43.396137 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:43.396117    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:43.397318 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:43.397293    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Aug 01 00:02:43.416005 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:43.415963    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" podStartSLOduration=2.457465275 podStartE2EDuration="25.415950366s" totalImagesPullingTime="19.266402025s" totalInitContainerRuntime="3.692083066s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:02:43.413918004 +0000 UTC m=+993.873380828" watchObservedRunningTime="2026-08-01 00:02:43.415950366 +0000 UTC m=+993.875413179"
Aug 01 00:02:44.399783 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:44.399742    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Aug 01 00:02:49.404686 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:49.404659    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:02:49.405371 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:49.405342    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Aug 01 00:02:59.405346 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:02:59.405306    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Aug 01 00:03:09.405846 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:03:09.405802    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Aug 01 00:03:19.406166 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:03:19.406126    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Aug 01 00:03:29.405760 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:03:29.405726    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Aug 01 00:03:39.405744 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:03:39.405705    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Aug 01 00:03:49.000938 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:03:49.000886    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:03:49.406200 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:03:49.406167    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Aug 01 00:03:59.406103 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:03:59.406076    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:04:06.871864 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:06.871789    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:08.397430 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:08.397400    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"]
Aug 01 00:04:08.397889 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:08.397615    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kserve-container" containerID="cri-o://0530081e195ebc9d6034c3ba7ff2ae949927721912cca5eeb85adc560396e635" gracePeriod=30
Aug 01 00:04:08.397889 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:08.397648    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kube-rbac-proxy" containerID="cri-o://d6668082cdb4354b432001a74a53b2ea506e5bc230cde5cbbff1e11dc6f41cdd" gracePeriod=30
Aug 01 00:04:08.419995 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:08.419961    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:08.739897 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:08.739811    2546 generic.go:356] "Generic (PLEG): container finished" podID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerID="d6668082cdb4354b432001a74a53b2ea506e5bc230cde5cbbff1e11dc6f41cdd" exitCode=2
Aug 01 00:04:08.739897 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:08.739872    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" event={"ID":"23036ad5-7ce0-4f38-965d-f401a118cd10","Type":"ContainerDied","Data":"d6668082cdb4354b432001a74a53b2ea506e5bc230cde5cbbff1e11dc6f41cdd"}
Aug 01 00:04:09.400706 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:09.400666    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Aug 01 00:04:09.406186 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:09.406156    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Aug 01 00:04:12.027113 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.027075    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:12.191833 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.191789    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:12.739835 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.739813    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:04:12.756867 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.756843    2546 generic.go:356] "Generic (PLEG): container finished" podID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerID="0530081e195ebc9d6034c3ba7ff2ae949927721912cca5eeb85adc560396e635" exitCode=0
Aug 01 00:04:12.756984 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.756893    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"
Aug 01 00:04:12.757028 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.756896    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" event={"ID":"23036ad5-7ce0-4f38-965d-f401a118cd10","Type":"ContainerDied","Data":"0530081e195ebc9d6034c3ba7ff2ae949927721912cca5eeb85adc560396e635"}
Aug 01 00:04:12.757069 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.757042    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw" event={"ID":"23036ad5-7ce0-4f38-965d-f401a118cd10","Type":"ContainerDied","Data":"70943cb860ccb2f1136e09e8af382106866880635eb69aad71d8b6f8e7b7547a"}
Aug 01 00:04:12.757104 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.757081    2546 scope.go:117] "RemoveContainer" containerID="d6668082cdb4354b432001a74a53b2ea506e5bc230cde5cbbff1e11dc6f41cdd"
Aug 01 00:04:12.765983 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.765964    2546 scope.go:117] "RemoveContainer" containerID="0530081e195ebc9d6034c3ba7ff2ae949927721912cca5eeb85adc560396e635"
Aug 01 00:04:12.772968 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.772951    2546 scope.go:117] "RemoveContainer" containerID="1e299b99ffe0097b52b4c121aff04ff8e13b309ae60572e63deef2a0f4ceecdb"
Aug 01 00:04:12.779945 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.779919    2546 scope.go:117] "RemoveContainer" containerID="d6668082cdb4354b432001a74a53b2ea506e5bc230cde5cbbff1e11dc6f41cdd"
Aug 01 00:04:12.780220 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:04:12.780202    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6668082cdb4354b432001a74a53b2ea506e5bc230cde5cbbff1e11dc6f41cdd\": container with ID starting with d6668082cdb4354b432001a74a53b2ea506e5bc230cde5cbbff1e11dc6f41cdd not found: ID does not exist" containerID="d6668082cdb4354b432001a74a53b2ea506e5bc230cde5cbbff1e11dc6f41cdd"
Aug 01 00:04:12.780285 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.780228    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6668082cdb4354b432001a74a53b2ea506e5bc230cde5cbbff1e11dc6f41cdd"} err="failed to get container status \"d6668082cdb4354b432001a74a53b2ea506e5bc230cde5cbbff1e11dc6f41cdd\": rpc error: code = NotFound desc = could not find container \"d6668082cdb4354b432001a74a53b2ea506e5bc230cde5cbbff1e11dc6f41cdd\": container with ID starting with d6668082cdb4354b432001a74a53b2ea506e5bc230cde5cbbff1e11dc6f41cdd not found: ID does not exist"
Aug 01 00:04:12.780285 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.780246    2546 scope.go:117] "RemoveContainer" containerID="0530081e195ebc9d6034c3ba7ff2ae949927721912cca5eeb85adc560396e635"
Aug 01 00:04:12.780547 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:04:12.780519    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0530081e195ebc9d6034c3ba7ff2ae949927721912cca5eeb85adc560396e635\": container with ID starting with 0530081e195ebc9d6034c3ba7ff2ae949927721912cca5eeb85adc560396e635 not found: ID does not exist" containerID="0530081e195ebc9d6034c3ba7ff2ae949927721912cca5eeb85adc560396e635"
Aug 01 00:04:12.780619 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.780546    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0530081e195ebc9d6034c3ba7ff2ae949927721912cca5eeb85adc560396e635"} err="failed to get container status \"0530081e195ebc9d6034c3ba7ff2ae949927721912cca5eeb85adc560396e635\": rpc error: code = NotFound desc = could not find container \"0530081e195ebc9d6034c3ba7ff2ae949927721912cca5eeb85adc560396e635\": container with ID starting with 0530081e195ebc9d6034c3ba7ff2ae949927721912cca5eeb85adc560396e635 not found: ID does not exist"
Aug 01 00:04:12.780619 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.780568    2546 scope.go:117] "RemoveContainer" containerID="1e299b99ffe0097b52b4c121aff04ff8e13b309ae60572e63deef2a0f4ceecdb"
Aug 01 00:04:12.780817 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:04:12.780802    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e299b99ffe0097b52b4c121aff04ff8e13b309ae60572e63deef2a0f4ceecdb\": container with ID starting with 1e299b99ffe0097b52b4c121aff04ff8e13b309ae60572e63deef2a0f4ceecdb not found: ID does not exist" containerID="1e299b99ffe0097b52b4c121aff04ff8e13b309ae60572e63deef2a0f4ceecdb"
Aug 01 00:04:12.780861 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.780825    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e299b99ffe0097b52b4c121aff04ff8e13b309ae60572e63deef2a0f4ceecdb"} err="failed to get container status \"1e299b99ffe0097b52b4c121aff04ff8e13b309ae60572e63deef2a0f4ceecdb\": rpc error: code = NotFound desc = could not find container \"1e299b99ffe0097b52b4c121aff04ff8e13b309ae60572e63deef2a0f4ceecdb\": container with ID starting with 1e299b99ffe0097b52b4c121aff04ff8e13b309ae60572e63deef2a0f4ceecdb not found: ID does not exist"
Aug 01 00:04:12.813876 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.813854    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23036ad5-7ce0-4f38-965d-f401a118cd10-kserve-provision-location\") pod \"23036ad5-7ce0-4f38-965d-f401a118cd10\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") "
Aug 01 00:04:12.813993 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.813909    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23036ad5-7ce0-4f38-965d-f401a118cd10-cabundle-cert\") pod \"23036ad5-7ce0-4f38-965d-f401a118cd10\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") "
Aug 01 00:04:12.813993 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.813963    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23036ad5-7ce0-4f38-965d-f401a118cd10-kube-rbac-proxy-sar-config\") pod \"23036ad5-7ce0-4f38-965d-f401a118cd10\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") "
Aug 01 00:04:12.814102 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.814000    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6ssq\" (UniqueName: \"kubernetes.io/projected/23036ad5-7ce0-4f38-965d-f401a118cd10-kube-api-access-q6ssq\") pod \"23036ad5-7ce0-4f38-965d-f401a118cd10\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") "
Aug 01 00:04:12.814102 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.814046    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23036ad5-7ce0-4f38-965d-f401a118cd10-proxy-tls\") pod \"23036ad5-7ce0-4f38-965d-f401a118cd10\" (UID: \"23036ad5-7ce0-4f38-965d-f401a118cd10\") "
Aug 01 00:04:12.814228 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.814203    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23036ad5-7ce0-4f38-965d-f401a118cd10-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "23036ad5-7ce0-4f38-965d-f401a118cd10" (UID: "23036ad5-7ce0-4f38-965d-f401a118cd10"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:04:12.814331 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.814302    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23036ad5-7ce0-4f38-965d-f401a118cd10-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "23036ad5-7ce0-4f38-965d-f401a118cd10" (UID: "23036ad5-7ce0-4f38-965d-f401a118cd10"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:04:12.814394 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.814324    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23036ad5-7ce0-4f38-965d-f401a118cd10-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "23036ad5-7ce0-4f38-965d-f401a118cd10" (UID: "23036ad5-7ce0-4f38-965d-f401a118cd10"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:04:12.815943 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.815923    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23036ad5-7ce0-4f38-965d-f401a118cd10-kube-api-access-q6ssq" (OuterVolumeSpecName: "kube-api-access-q6ssq") pod "23036ad5-7ce0-4f38-965d-f401a118cd10" (UID: "23036ad5-7ce0-4f38-965d-f401a118cd10"). InnerVolumeSpecName "kube-api-access-q6ssq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:04:12.816114 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.816095    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23036ad5-7ce0-4f38-965d-f401a118cd10-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "23036ad5-7ce0-4f38-965d-f401a118cd10" (UID: "23036ad5-7ce0-4f38-965d-f401a118cd10"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:04:12.914660 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.914639    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6ssq\" (UniqueName: \"kubernetes.io/projected/23036ad5-7ce0-4f38-965d-f401a118cd10-kube-api-access-q6ssq\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:04:12.914660 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.914659    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23036ad5-7ce0-4f38-965d-f401a118cd10-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:04:12.914821 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.914669    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23036ad5-7ce0-4f38-965d-f401a118cd10-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:04:12.914821 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.914677    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23036ad5-7ce0-4f38-965d-f401a118cd10-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:04:12.914821 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:12.914686    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23036ad5-7ce0-4f38-965d-f401a118cd10-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:04:13.084065 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.084041    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"]
Aug 01 00:04:13.090371 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.090348    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bnvcw"]
Aug 01 00:04:13.527958 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.527922    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:13.548112 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.548084    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"]
Aug 01 00:04:13.548437 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.548422    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="storage-initializer"
Aug 01 00:04:13.548437 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.548438    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="storage-initializer"
Aug 01 00:04:13.548554 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.548452    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kube-rbac-proxy"
Aug 01 00:04:13.548554 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.548461    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kube-rbac-proxy"
Aug 01 00:04:13.548554 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.548474    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kserve-container"
Aug 01 00:04:13.548554 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.548480    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kserve-container"
Aug 01 00:04:13.548554 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.548541    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kube-rbac-proxy"
Aug 01 00:04:13.548554 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.548551    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" containerName="kserve-container"
Aug 01 00:04:13.552941 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.552922    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.555941 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.555920    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Aug 01 00:04:13.556048 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.555945    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:04:13.556048 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.555946    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Aug 01 00:04:13.556048 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.556026    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:04:13.556048 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.555947    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:04:13.556287 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.556056    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:04:13.556445 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.556428    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:04:13.564428 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.564404    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"]
Aug 01 00:04:13.620739 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.620709    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.620919 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.620757    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp7q6\" (UniqueName: \"kubernetes.io/projected/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kube-api-access-fp7q6\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.620919 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.620779    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.620919 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.620864    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.620919 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.620900    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.664513 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.664485    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:13.721628 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.721597    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.721787 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.721637    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.721787 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.721671    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.721787 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.721709    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp7q6\" (UniqueName: \"kubernetes.io/projected/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kube-api-access-fp7q6\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.721787 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.721728    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.722115 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.722093    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.722310 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.722292    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.722310 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.722304    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.724134 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.724116    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.730751 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.730724    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp7q6\" (UniqueName: \"kubernetes.io/projected/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kube-api-access-fp7q6\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.862801 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.862777    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:13.999770 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:13.999746    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"]
Aug 01 00:04:14.002300 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:04:14.002269    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495e40b4_c6d6_4e6c_a8dd_e2a81bb7f2f0.slice/crio-f87576b5c30c3881951d1874c24ad2368c1bd9ef3a695652b6fef142883ac783 WatchSource:0}: Error finding container f87576b5c30c3881951d1874c24ad2368c1bd9ef3a695652b6fef142883ac783: Status 404 returned error can't find the container with id f87576b5c30c3881951d1874c24ad2368c1bd9ef3a695652b6fef142883ac783
Aug 01 00:04:14.079482 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:14.077657    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23036ad5-7ce0-4f38-965d-f401a118cd10" path="/var/lib/kubelet/pods/23036ad5-7ce0-4f38-965d-f401a118cd10/volumes"
Aug 01 00:04:14.767496 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:14.767465    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" event={"ID":"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0","Type":"ContainerStarted","Data":"7958777e96ab6e535945dea476ffd214b89a4a86654d5ba0a81c8f983eb7209f"}
Aug 01 00:04:14.767909 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:14.767523    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" event={"ID":"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0","Type":"ContainerStarted","Data":"f87576b5c30c3881951d1874c24ad2368c1bd9ef3a695652b6fef142883ac783"}
Aug 01 00:04:17.780766 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:17.780732    2546 generic.go:356] "Generic (PLEG): container finished" podID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerID="7958777e96ab6e535945dea476ffd214b89a4a86654d5ba0a81c8f983eb7209f" exitCode=0
Aug 01 00:04:17.781156 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:17.780807    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" event={"ID":"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0","Type":"ContainerDied","Data":"7958777e96ab6e535945dea476ffd214b89a4a86654d5ba0a81c8f983eb7209f"}
Aug 01 00:04:18.787475 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:18.787442    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" event={"ID":"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0","Type":"ContainerStarted","Data":"df33abad43b2af7f54603491de7839931cd2eb25e5e540e47ff48844d2877096"}
Aug 01 00:04:18.787846 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:18.787505    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" event={"ID":"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0","Type":"ContainerStarted","Data":"8e9d857370ca85d17e656a96c33babe0e4bce73d487f6d822c98e4f705dc66e5"}
Aug 01 00:04:18.787846 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:18.787718    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:18.807409 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:18.807367    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" podStartSLOduration=2.211203372 podStartE2EDuration="5.807351321s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.596147949s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:04:18.805882012 +0000 UTC m=+1089.265344824" watchObservedRunningTime="2026-08-01 00:04:18.807351321 +0000 UTC m=+1089.266814134"
Aug 01 00:04:19.791550 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:19.791518    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:19.793058 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:19.793030    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:04:20.795025 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:20.794983    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:04:25.799683 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:25.799654    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:04:25.800223 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:25.800194    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:04:29.980793 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:29.980760    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:04:35.800135 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:35.800090    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:04:45.800896 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:45.800861    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:04:55.800856 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:04:55.800813    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:05:05.800148 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:05.800106    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:05:15.800358 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:15.800319    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:05:25.703814 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:25.703780    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:25.762345 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:25.762305    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:25.800869 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:25.800839    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:05:35.801627 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:35.801592    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:05:43.690899 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:43.690867    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"]
Aug 01 00:05:43.691412 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:43.691094    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kserve-container" containerID="cri-o://8e9d857370ca85d17e656a96c33babe0e4bce73d487f6d822c98e4f705dc66e5" gracePeriod=30
Aug 01 00:05:43.691412 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:43.691136    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kube-rbac-proxy" containerID="cri-o://df33abad43b2af7f54603491de7839931cd2eb25e5e540e47ff48844d2877096" gracePeriod=30
Aug 01 00:05:43.711129 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:43.711101    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:44.133746 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:44.133714    2546 generic.go:356] "Generic (PLEG): container finished" podID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerID="df33abad43b2af7f54603491de7839931cd2eb25e5e540e47ff48844d2877096" exitCode=2
Aug 01 00:05:44.133903 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:44.133786    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" event={"ID":"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0","Type":"ContainerDied","Data":"df33abad43b2af7f54603491de7839931cd2eb25e5e540e47ff48844d2877096"}
Aug 01 00:05:45.795519 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:45.795479    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Aug 01 00:05:45.800715 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:45.800679    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Aug 01 00:05:48.730876 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.730848    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:05:48.809985 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.809949    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:48.869092 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.869057    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"]
Aug 01 00:05:48.869494 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.869477    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="storage-initializer"
Aug 01 00:05:48.869656 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.869496    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="storage-initializer"
Aug 01 00:05:48.869656 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.869507    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kube-rbac-proxy"
Aug 01 00:05:48.869656 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.869516    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kube-rbac-proxy"
Aug 01 00:05:48.869656 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.869527    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kserve-container"
Aug 01 00:05:48.869656 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.869535    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kserve-container"
Aug 01 00:05:48.869656 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.869624    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kserve-container"
Aug 01 00:05:48.869656 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.869637    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerName="kube-rbac-proxy"
Aug 01 00:05:48.872698 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.872678    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:48.875080 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.875063    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Aug 01 00:05:48.875154 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.875100    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Aug 01 00:05:48.882351 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.882331    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"]
Aug 01 00:05:48.893516 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.893491    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-proxy-tls\") pod \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") "
Aug 01 00:05:48.893614 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.893547    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-cabundle-cert\") pod \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") "
Aug 01 00:05:48.893614 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.893598    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp7q6\" (UniqueName: \"kubernetes.io/projected/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kube-api-access-fp7q6\") pod \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") "
Aug 01 00:05:48.893707 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.893687    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kserve-provision-location\") pod \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") "
Aug 01 00:05:48.893764 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.893730    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kube-rbac-proxy-sar-config\") pod \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\" (UID: \"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0\") "
Aug 01 00:05:48.894028 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.894002    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" (UID: "495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:05:48.894131 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.893932    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" (UID: "495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:05:48.894131 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.894067    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" (UID: "495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:05:48.895537 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.895519    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" (UID: "495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:05:48.895652 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.895636    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kube-api-access-fp7q6" (OuterVolumeSpecName: "kube-api-access-fp7q6") pod "495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" (UID: "495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0"). InnerVolumeSpecName "kube-api-access-fp7q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:05:48.904449 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.904424    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:05:48.995046 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.995009    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:48.995199 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.995049    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl8mw\" (UniqueName: \"kubernetes.io/projected/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kube-api-access-kl8mw\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:48.995199 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.995091    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:48.995199 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.995119    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:48.995199 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.995194    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:48.995436 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.995301    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:05:48.995436 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.995319    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:05:48.995436 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.995335    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:05:48.995436 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.995349    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:05:48.995436 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:48.995361    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp7q6\" (UniqueName: \"kubernetes.io/projected/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0-kube-api-access-fp7q6\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:05:49.096686 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.096652    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:49.096833 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.096690    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl8mw\" (UniqueName: \"kubernetes.io/projected/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kube-api-access-kl8mw\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:49.096833 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.096812    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:49.096968 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.096854    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:49.096968 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.096927    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:49.097226 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.097205    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:49.097336 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.097298    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:49.097397 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.097380    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:49.099293 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.099274    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:49.104559 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.104534    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl8mw\" (UniqueName: \"kubernetes.io/projected/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kube-api-access-kl8mw\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:49.155168 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.155105    2546 generic.go:356] "Generic (PLEG): container finished" podID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" containerID="8e9d857370ca85d17e656a96c33babe0e4bce73d487f6d822c98e4f705dc66e5" exitCode=0
Aug 01 00:05:49.155279 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.155149    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" event={"ID":"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0","Type":"ContainerDied","Data":"8e9d857370ca85d17e656a96c33babe0e4bce73d487f6d822c98e4f705dc66e5"}
Aug 01 00:05:49.155279 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.155171    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"
Aug 01 00:05:49.155279 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.155215    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs" event={"ID":"495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0","Type":"ContainerDied","Data":"f87576b5c30c3881951d1874c24ad2368c1bd9ef3a695652b6fef142883ac783"}
Aug 01 00:05:49.155279 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.155241    2546 scope.go:117] "RemoveContainer" containerID="df33abad43b2af7f54603491de7839931cd2eb25e5e540e47ff48844d2877096"
Aug 01 00:05:49.163836 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.163814    2546 scope.go:117] "RemoveContainer" containerID="8e9d857370ca85d17e656a96c33babe0e4bce73d487f6d822c98e4f705dc66e5"
Aug 01 00:05:49.171125 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.171110    2546 scope.go:117] "RemoveContainer" containerID="7958777e96ab6e535945dea476ffd214b89a4a86654d5ba0a81c8f983eb7209f"
Aug 01 00:05:49.178440 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.178415    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"]
Aug 01 00:05:49.179628 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.179611    2546 scope.go:117] "RemoveContainer" containerID="df33abad43b2af7f54603491de7839931cd2eb25e5e540e47ff48844d2877096"
Aug 01 00:05:49.179892 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:05:49.179870    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df33abad43b2af7f54603491de7839931cd2eb25e5e540e47ff48844d2877096\": container with ID starting with df33abad43b2af7f54603491de7839931cd2eb25e5e540e47ff48844d2877096 not found: ID does not exist" containerID="df33abad43b2af7f54603491de7839931cd2eb25e5e540e47ff48844d2877096"
Aug 01 00:05:49.179995 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.179897    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df33abad43b2af7f54603491de7839931cd2eb25e5e540e47ff48844d2877096"} err="failed to get container status \"df33abad43b2af7f54603491de7839931cd2eb25e5e540e47ff48844d2877096\": rpc error: code = NotFound desc = could not find container \"df33abad43b2af7f54603491de7839931cd2eb25e5e540e47ff48844d2877096\": container with ID starting with df33abad43b2af7f54603491de7839931cd2eb25e5e540e47ff48844d2877096 not found: ID does not exist"
Aug 01 00:05:49.179995 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.179920    2546 scope.go:117] "RemoveContainer" containerID="8e9d857370ca85d17e656a96c33babe0e4bce73d487f6d822c98e4f705dc66e5"
Aug 01 00:05:49.180146 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:05:49.180131    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e9d857370ca85d17e656a96c33babe0e4bce73d487f6d822c98e4f705dc66e5\": container with ID starting with 8e9d857370ca85d17e656a96c33babe0e4bce73d487f6d822c98e4f705dc66e5 not found: ID does not exist" containerID="8e9d857370ca85d17e656a96c33babe0e4bce73d487f6d822c98e4f705dc66e5"
Aug 01 00:05:49.180210 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.180149    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9d857370ca85d17e656a96c33babe0e4bce73d487f6d822c98e4f705dc66e5"} err="failed to get container status \"8e9d857370ca85d17e656a96c33babe0e4bce73d487f6d822c98e4f705dc66e5\": rpc error: code = NotFound desc = could not find container \"8e9d857370ca85d17e656a96c33babe0e4bce73d487f6d822c98e4f705dc66e5\": container with ID starting with 8e9d857370ca85d17e656a96c33babe0e4bce73d487f6d822c98e4f705dc66e5 not found: ID does not exist"
Aug 01 00:05:49.180210 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.180161    2546 scope.go:117] "RemoveContainer" containerID="7958777e96ab6e535945dea476ffd214b89a4a86654d5ba0a81c8f983eb7209f"
Aug 01 00:05:49.180455 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:05:49.180436    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7958777e96ab6e535945dea476ffd214b89a4a86654d5ba0a81c8f983eb7209f\": container with ID starting with 7958777e96ab6e535945dea476ffd214b89a4a86654d5ba0a81c8f983eb7209f not found: ID does not exist" containerID="7958777e96ab6e535945dea476ffd214b89a4a86654d5ba0a81c8f983eb7209f"
Aug 01 00:05:49.180516 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.180459    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7958777e96ab6e535945dea476ffd214b89a4a86654d5ba0a81c8f983eb7209f"} err="failed to get container status \"7958777e96ab6e535945dea476ffd214b89a4a86654d5ba0a81c8f983eb7209f\": rpc error: code = NotFound desc = could not find container \"7958777e96ab6e535945dea476ffd214b89a4a86654d5ba0a81c8f983eb7209f\": container with ID starting with 7958777e96ab6e535945dea476ffd214b89a4a86654d5ba0a81c8f983eb7209f not found: ID does not exist"
Aug 01 00:05:49.181125 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.181105    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4v4rs"]
Aug 01 00:05:49.182645 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.182631    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:49.319995 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:49.319968    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"]
Aug 01 00:05:49.321808 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:05:49.321780    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d0b07d_cfee_4d3c_90a4_3ceab4bf185f.slice/crio-3c5efeb53d93aa37214b21576d42341a2df67683e4ffb4637141339e25d20a35 WatchSource:0}: Error finding container 3c5efeb53d93aa37214b21576d42341a2df67683e4ffb4637141339e25d20a35: Status 404 returned error can't find the container with id 3c5efeb53d93aa37214b21576d42341a2df67683e4ffb4637141339e25d20a35
Aug 01 00:05:50.077412 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:50.077382    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0" path="/var/lib/kubelet/pods/495e40b4-c6d6-4e6c-a8dd-e2a81bb7f2f0/volumes"
Aug 01 00:05:50.160854 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:50.160815    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" event={"ID":"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f","Type":"ContainerStarted","Data":"27b345fc663f1d8e2da19ef83480a1e7d60c004d2239fb1b97dca2f2e34d230e"}
Aug 01 00:05:50.161045 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:50.160882    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" event={"ID":"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f","Type":"ContainerStarted","Data":"3c5efeb53d93aa37214b21576d42341a2df67683e4ffb4637141339e25d20a35"}
Aug 01 00:05:53.174686 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:53.174655    2546 generic.go:356] "Generic (PLEG): container finished" podID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerID="27b345fc663f1d8e2da19ef83480a1e7d60c004d2239fb1b97dca2f2e34d230e" exitCode=0
Aug 01 00:05:53.175050 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:53.174728    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" event={"ID":"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f","Type":"ContainerDied","Data":"27b345fc663f1d8e2da19ef83480a1e7d60c004d2239fb1b97dca2f2e34d230e"}
Aug 01 00:05:54.180771 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:54.180731    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" event={"ID":"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f","Type":"ContainerStarted","Data":"da5ae4fb898085166d669d1a71e0e06f92f6aec74c2140b6abd03c36c3e34251"}
Aug 01 00:05:54.181215 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:54.180796    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" event={"ID":"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f","Type":"ContainerStarted","Data":"42532c83c4f1a153cf51a9235fa2834c14fbee6c912e2ae052fb4a4f5357a66e"}
Aug 01 00:05:54.181215 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:54.180982    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:05:54.205802 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:54.205755    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" podStartSLOduration=2.45312329 podStartE2EDuration="6.205742893s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.752619603s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:05:54.203970956 +0000 UTC m=+1184.663433790" watchObservedRunningTime="2026-08-01 00:05:54.205742893 +0000 UTC m=+1184.665205706"
Aug 01 00:05:55.184389 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:05:55.184362    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:06:01.194044 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:01.194013    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:06:28.336833 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:28.336797    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:31.195121 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:31.195083    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.38:8080: connect: connection refused"
Aug 01 00:06:33.483511 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.483461    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:33.532053 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.532016    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"]
Aug 01 00:06:33.535607 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.535584    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.538304 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.538283    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Aug 01 00:06:33.538426 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.538288    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Aug 01 00:06:33.538982 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.538957    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"]
Aug 01 00:06:33.645359 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.645333    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:06:33.655288 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.655267    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.655406 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.655309    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.655406 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.655348    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.655493 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.655407    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6bdp\" (UniqueName: \"kubernetes.io/projected/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kube-api-access-x6bdp\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.655493 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.655465    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.756208 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.756127    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.756208 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.756178    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.756432 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.756214    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.756432 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.756273    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6bdp\" (UniqueName: \"kubernetes.io/projected/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kube-api-access-x6bdp\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.756432 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.756419    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.756803 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.756771    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.756882 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.756837    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.756950 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.756916    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.758679 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.758659    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.766625 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.766607    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6bdp\" (UniqueName: \"kubernetes.io/projected/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kube-api-access-x6bdp\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.846445 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.846407    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:06:33.979615 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:33.979577    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"]
Aug 01 00:06:33.982551 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:06:33.982524    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c1ca2d_08b5_4fce_92b2_f3f1d3b5983b.slice/crio-f28bd7de81db76b27b21f38a9d2c865eacbc512913d4b89474b2a7cc3792cf65 WatchSource:0}: Error finding container f28bd7de81db76b27b21f38a9d2c865eacbc512913d4b89474b2a7cc3792cf65: Status 404 returned error can't find the container with id f28bd7de81db76b27b21f38a9d2c865eacbc512913d4b89474b2a7cc3792cf65
Aug 01 00:06:34.345453 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:34.345356    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f" event={"ID":"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b","Type":"ContainerStarted","Data":"942695f33b7f73580d1537a6f84474adc03798f8501d1c021f038d3add2c8eee"}
Aug 01 00:06:34.345627 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:34.345469    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f" event={"ID":"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b","Type":"ContainerStarted","Data":"f28bd7de81db76b27b21f38a9d2c865eacbc512913d4b89474b2a7cc3792cf65"}
Aug 01 00:06:41.195438 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:41.195401    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.38:8080: connect: connection refused"
Aug 01 00:06:51.194776 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:06:51.194731    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.38:8080: connect: connection refused"
Aug 01 00:07:01.195506 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:01.195464    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.38:8080: connect: connection refused"
Aug 01 00:07:03.464060 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:03.464025    2546 generic.go:356] "Generic (PLEG): container finished" podID="64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" containerID="942695f33b7f73580d1537a6f84474adc03798f8501d1c021f038d3add2c8eee" exitCode=0
Aug 01 00:07:03.464060 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:03.464067    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f" event={"ID":"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b","Type":"ContainerDied","Data":"942695f33b7f73580d1537a6f84474adc03798f8501d1c021f038d3add2c8eee"}
Aug 01 00:07:11.221115 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:11.221083    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:07:16.706018 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:16.705981    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:19.854768 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:19.854724    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"]
Aug 01 00:07:19.855209 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:19.855005    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="kserve-container" containerID="cri-o://42532c83c4f1a153cf51a9235fa2834c14fbee6c912e2ae052fb4a4f5357a66e" gracePeriod=30
Aug 01 00:07:19.855552 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:19.855415    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="kube-rbac-proxy" containerID="cri-o://da5ae4fb898085166d669d1a71e0e06f92f6aec74c2140b6abd03c36c3e34251" gracePeriod=30
Aug 01 00:07:19.875689 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:19.875659    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:20.549187 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:20.549151    2546 generic.go:356] "Generic (PLEG): container finished" podID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerID="da5ae4fb898085166d669d1a71e0e06f92f6aec74c2140b6abd03c36c3e34251" exitCode=2
Aug 01 00:07:20.549388 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:20.549196    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" event={"ID":"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f","Type":"ContainerDied","Data":"da5ae4fb898085166d669d1a71e0e06f92f6aec74c2140b6abd03c36c3e34251"}
Aug 01 00:07:21.071371 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:21.071323    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.38:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.38:8080: connect: connection refused"
Aug 01 00:07:21.188924 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:21.188881    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Aug 01 00:07:24.569787 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:24.569755    2546 generic.go:356] "Generic (PLEG): container finished" podID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerID="42532c83c4f1a153cf51a9235fa2834c14fbee6c912e2ae052fb4a4f5357a66e" exitCode=0
Aug 01 00:07:24.570181 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:24.569837    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" event={"ID":"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f","Type":"ContainerDied","Data":"42532c83c4f1a153cf51a9235fa2834c14fbee6c912e2ae052fb4a4f5357a66e"}
Aug 01 00:07:25.000648 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.000616    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:25.137472 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.137436    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:07:25.185981 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.185944    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kl8mw\" (UniqueName: \"kubernetes.io/projected/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kube-api-access-kl8mw\") pod \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") "
Aug 01 00:07:25.186153 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.186049    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-cabundle-cert\") pod \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") "
Aug 01 00:07:25.186153 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.186094    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-proxy-tls\") pod \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") "
Aug 01 00:07:25.186153 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.186138    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kserve-provision-location\") pod \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") "
Aug 01 00:07:25.186355 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.186185    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kube-rbac-proxy-sar-config\") pod \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\" (UID: \"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f\") "
Aug 01 00:07:25.186615 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.186514    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" (UID: "13d0b07d-cfee-4d3c-90a4-3ceab4bf185f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:07:25.186711 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.186506    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" (UID: "13d0b07d-cfee-4d3c-90a4-3ceab4bf185f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:07:25.186711 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.186682    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" (UID: "13d0b07d-cfee-4d3c-90a4-3ceab4bf185f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:07:25.188610 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.188570    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" (UID: "13d0b07d-cfee-4d3c-90a4-3ceab4bf185f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:07:25.188742 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.188704    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kube-api-access-kl8mw" (OuterVolumeSpecName: "kube-api-access-kl8mw") pod "13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" (UID: "13d0b07d-cfee-4d3c-90a4-3ceab4bf185f"). InnerVolumeSpecName "kube-api-access-kl8mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:07:25.287799 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.287705    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:07:25.287799 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.287744    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:07:25.287799 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.287760    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:07:25.287799 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.287773    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl8mw\" (UniqueName: \"kubernetes.io/projected/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-kube-api-access-kl8mw\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:07:25.287799 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.287788    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:07:25.577291 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.577195    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"
Aug 01 00:07:25.577291 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.577201    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8" event={"ID":"13d0b07d-cfee-4d3c-90a4-3ceab4bf185f","Type":"ContainerDied","Data":"3c5efeb53d93aa37214b21576d42341a2df67683e4ffb4637141339e25d20a35"}
Aug 01 00:07:25.577779 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.577333    2546 scope.go:117] "RemoveContainer" containerID="da5ae4fb898085166d669d1a71e0e06f92f6aec74c2140b6abd03c36c3e34251"
Aug 01 00:07:25.578603 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.578547    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:07:25.587308 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.587285    2546 scope.go:117] "RemoveContainer" containerID="42532c83c4f1a153cf51a9235fa2834c14fbee6c912e2ae052fb4a4f5357a66e"
Aug 01 00:07:25.596338 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.596320    2546 scope.go:117] "RemoveContainer" containerID="27b345fc663f1d8e2da19ef83480a1e7d60c004d2239fb1b97dca2f2e34d230e"
Aug 01 00:07:25.601568 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.601547    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"]
Aug 01 00:07:25.604274 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:25.604239    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8j5b8"]
Aug 01 00:07:26.076953 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:07:26.076914    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" path="/var/lib/kubelet/pods/13d0b07d-cfee-4d3c-90a4-3ceab4bf185f/volumes"
Aug 01 00:08:55.188292 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:08:55.188223    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:00.005560 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:09:00.005525    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f" event={"ID":"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b","Type":"ContainerStarted","Data":"3ec9812c28059c24a6afa524a5757c9cdd84adcafb86407c4f8dbe1c62fb992c"}
Aug 01 00:09:00.309599 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:09:00.309555    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:00.861224 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:09:00.861193    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:09:01.011719 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:09:01.011680    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f" event={"ID":"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b","Type":"ContainerStarted","Data":"9548c7f11add71a524d78e12c5c46f4804e8e4697113c531da391eadf90ab3d9"}
Aug 01 00:09:01.012093 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:09:01.011809    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:09:01.033438 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:09:01.033380    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f" podStartSLOduration=2.966044137 podStartE2EDuration="2m28.03336042s" totalImagesPullingTime="1m56.358001655s" totalInitContainerRuntime="28.709314628s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:09:01.029968781 +0000 UTC m=+1371.489431595" watchObservedRunningTime="2026-08-01 00:09:01.03336042 +0000 UTC m=+1371.492823234"
Aug 01 00:09:02.016090 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:09:02.016055    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:09:08.025977 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:09:08.025941    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:10:30.533769 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:30.533682    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:34.075245 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:34.075220    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:10:35.587927 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:35.587895    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:35.881013 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:35.880972    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:37.654971 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:37.654941    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:42.758075 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.758046    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:42.783912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.783881    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"]
Aug 01 00:10:42.784221 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.784210    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="kube-rbac-proxy"
Aug 01 00:10:42.784301 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.784223    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="kube-rbac-proxy"
Aug 01 00:10:42.784301 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.784238    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="storage-initializer"
Aug 01 00:10:42.784301 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.784243    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="storage-initializer"
Aug 01 00:10:42.784301 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.784287    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="kserve-container"
Aug 01 00:10:42.784301 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.784294    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="kserve-container"
Aug 01 00:10:42.784454 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.784357    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="kube-rbac-proxy"
Aug 01 00:10:42.784454 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.784368    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d0b07d-cfee-4d3c-90a4-3ceab4bf185f" containerName="kserve-container"
Aug 01 00:10:42.786535 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.786520    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:42.788819 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.788789    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Aug 01 00:10:42.788819 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.788810    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Aug 01 00:10:42.798582 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.798555    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"]
Aug 01 00:10:42.960202 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.960161    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a24215b1-896e-4628-8359-101e45eb32d3-proxy-tls\") pod \"isvc-sklearn-predictor-868cb56654-2dpgb\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:42.960202 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.960204    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a24215b1-896e-4628-8359-101e45eb32d3-cabundle-cert\") pod \"isvc-sklearn-predictor-868cb56654-2dpgb\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:42.960456 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.960237    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a24215b1-896e-4628-8359-101e45eb32d3-kserve-provision-location\") pod \"isvc-sklearn-predictor-868cb56654-2dpgb\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:42.960456 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.960308    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a24215b1-896e-4628-8359-101e45eb32d3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-868cb56654-2dpgb\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:42.960456 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:42.960366    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqt7f\" (UniqueName: \"kubernetes.io/projected/a24215b1-896e-4628-8359-101e45eb32d3-kube-api-access-hqt7f\") pod \"isvc-sklearn-predictor-868cb56654-2dpgb\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:43.013703 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:43.013627    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:43.061335 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:43.061301    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqt7f\" (UniqueName: \"kubernetes.io/projected/a24215b1-896e-4628-8359-101e45eb32d3-kube-api-access-hqt7f\") pod \"isvc-sklearn-predictor-868cb56654-2dpgb\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:43.061505 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:43.061371    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a24215b1-896e-4628-8359-101e45eb32d3-proxy-tls\") pod \"isvc-sklearn-predictor-868cb56654-2dpgb\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:43.061505 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:43.061402    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a24215b1-896e-4628-8359-101e45eb32d3-cabundle-cert\") pod \"isvc-sklearn-predictor-868cb56654-2dpgb\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:43.061505 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:43.061428    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a24215b1-896e-4628-8359-101e45eb32d3-kserve-provision-location\") pod \"isvc-sklearn-predictor-868cb56654-2dpgb\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:43.061505 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:43.061453    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a24215b1-896e-4628-8359-101e45eb32d3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-868cb56654-2dpgb\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:43.061882 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:43.061859    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a24215b1-896e-4628-8359-101e45eb32d3-kserve-provision-location\") pod \"isvc-sklearn-predictor-868cb56654-2dpgb\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:43.062312 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:43.062284    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a24215b1-896e-4628-8359-101e45eb32d3-cabundle-cert\") pod \"isvc-sklearn-predictor-868cb56654-2dpgb\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:43.062411 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:43.062284    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a24215b1-896e-4628-8359-101e45eb32d3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-868cb56654-2dpgb\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:43.063858 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:43.063832    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a24215b1-896e-4628-8359-101e45eb32d3-proxy-tls\") pod \"isvc-sklearn-predictor-868cb56654-2dpgb\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:43.070490 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:43.070466    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqt7f\" (UniqueName: \"kubernetes.io/projected/a24215b1-896e-4628-8359-101e45eb32d3-kube-api-access-hqt7f\") pod \"isvc-sklearn-predictor-868cb56654-2dpgb\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:43.096947 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:43.096921    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:43.227800 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:43.227572    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"]
Aug 01 00:10:43.230169 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:10:43.230139    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24215b1_896e_4628_8359_101e45eb32d3.slice/crio-a13a1e68bf3da15c308127ecfcf88db4ea4da078b3687b0b0d19f70003ca1ea9 WatchSource:0}: Error finding container a13a1e68bf3da15c308127ecfcf88db4ea4da078b3687b0b0d19f70003ca1ea9: Status 404 returned error can't find the container with id a13a1e68bf3da15c308127ecfcf88db4ea4da078b3687b0b0d19f70003ca1ea9
Aug 01 00:10:43.417332 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:43.417290    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" event={"ID":"a24215b1-896e-4628-8359-101e45eb32d3","Type":"ContainerStarted","Data":"6d0ce798d4f2210a0b72da4345768c7a2825a2d401770987f330c05a1cf32a36"}
Aug 01 00:10:43.417493 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:43.417369    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" event={"ID":"a24215b1-896e-4628-8359-101e45eb32d3","Type":"ContainerStarted","Data":"a13a1e68bf3da15c308127ecfcf88db4ea4da078b3687b0b0d19f70003ca1ea9"}
Aug 01 00:10:45.111479 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:45.111431    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f" podUID="64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.39:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Aug 01 00:10:47.435999 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:47.435960    2546 generic.go:356] "Generic (PLEG): container finished" podID="a24215b1-896e-4628-8359-101e45eb32d3" containerID="6d0ce798d4f2210a0b72da4345768c7a2825a2d401770987f330c05a1cf32a36" exitCode=0
Aug 01 00:10:47.436392 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:47.436013    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" event={"ID":"a24215b1-896e-4628-8359-101e45eb32d3","Type":"ContainerDied","Data":"6d0ce798d4f2210a0b72da4345768c7a2825a2d401770987f330c05a1cf32a36"}
Aug 01 00:10:48.442850 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:48.442816    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" event={"ID":"a24215b1-896e-4628-8359-101e45eb32d3","Type":"ContainerStarted","Data":"62581100141f1c59a51deea771c0d9564d986add1a1e2f120a587644b29da8ba"}
Aug 01 00:10:48.443372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:48.442887    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" event={"ID":"a24215b1-896e-4628-8359-101e45eb32d3","Type":"ContainerStarted","Data":"66fa1f0171c35bdee7214f4b08bf6b72e5081839b77adba13351209921c79c32"}
Aug 01 00:10:48.443372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:48.443096    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:48.461336 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:48.461291    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" podStartSLOduration=2.82877199 podStartE2EDuration="6.461274535s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.632502545s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:10:48.460018202 +0000 UTC m=+1478.919481039" watchObservedRunningTime="2026-08-01 00:10:48.461274535 +0000 UTC m=+1478.920737347"
Aug 01 00:10:49.447007 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:49.446974    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:49.448400 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:49.448357    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:10:50.319140 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:50.319102    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"]
Aug 01 00:10:50.319373 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:50.319345    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f" podUID="64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" containerName="kserve-container" containerID="cri-o://3ec9812c28059c24a6afa524a5757c9cdd84adcafb86407c4f8dbe1c62fb992c" gracePeriod=30
Aug 01 00:10:50.319465 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:50.319404    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f" podUID="64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" containerName="kube-rbac-proxy" containerID="cri-o://9548c7f11add71a524d78e12c5c46f4804e8e4697113c531da391eadf90ab3d9" gracePeriod=30
Aug 01 00:10:50.342571 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:50.342544    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:50.452611 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:50.452580    2546 generic.go:356] "Generic (PLEG): container finished" podID="64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" containerID="9548c7f11add71a524d78e12c5c46f4804e8e4697113c531da391eadf90ab3d9" exitCode=2
Aug 01 00:10:50.452970 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:50.452648    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f" event={"ID":"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b","Type":"ContainerDied","Data":"9548c7f11add71a524d78e12c5c46f4804e8e4697113c531da391eadf90ab3d9"}
Aug 01 00:10:50.453071 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:50.453045    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:10:51.667757 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.667736    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:10:51.840445 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.840359    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-proxy-tls\") pod \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") "
Aug 01 00:10:51.840445 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.840419    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kserve-provision-location\") pod \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") "
Aug 01 00:10:51.840636 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.840534    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-cabundle-cert\") pod \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") "
Aug 01 00:10:51.840636 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.840568    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6bdp\" (UniqueName: \"kubernetes.io/projected/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kube-api-access-x6bdp\") pod \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") "
Aug 01 00:10:51.840717 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.840636    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kube-rbac-proxy-sar-config\") pod \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\" (UID: \"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b\") "
Aug 01 00:10:51.840984 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.840956    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" (UID: "64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:10:51.841093 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.841063    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" (UID: "64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:10:51.842615 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.842596    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kube-api-access-x6bdp" (OuterVolumeSpecName: "kube-api-access-x6bdp") pod "64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" (UID: "64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b"). InnerVolumeSpecName "kube-api-access-x6bdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:10:51.842801 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.842784    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" (UID: "64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:10:51.844505 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.844480    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" (UID: "64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:10:51.941338 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.941309    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:10:51.941338 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.941332    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:10:51.941338 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.941342    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:10:51.941534 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.941351    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6bdp\" (UniqueName: \"kubernetes.io/projected/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kube-api-access-x6bdp\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:10:51.941534 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:51.941360    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:10:52.466612 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:52.466578    2546 generic.go:356] "Generic (PLEG): container finished" podID="64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" containerID="3ec9812c28059c24a6afa524a5757c9cdd84adcafb86407c4f8dbe1c62fb992c" exitCode=0
Aug 01 00:10:52.466778 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:52.466635    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"
Aug 01 00:10:52.466778 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:52.466645    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f" event={"ID":"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b","Type":"ContainerDied","Data":"3ec9812c28059c24a6afa524a5757c9cdd84adcafb86407c4f8dbe1c62fb992c"}
Aug 01 00:10:52.466778 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:52.466692    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f" event={"ID":"64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b","Type":"ContainerDied","Data":"f28bd7de81db76b27b21f38a9d2c865eacbc512913d4b89474b2a7cc3792cf65"}
Aug 01 00:10:52.466778 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:52.466718    2546 scope.go:117] "RemoveContainer" containerID="9548c7f11add71a524d78e12c5c46f4804e8e4697113c531da391eadf90ab3d9"
Aug 01 00:10:52.474597 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:52.474578    2546 scope.go:117] "RemoveContainer" containerID="3ec9812c28059c24a6afa524a5757c9cdd84adcafb86407c4f8dbe1c62fb992c"
Aug 01 00:10:52.481630 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:52.481613    2546 scope.go:117] "RemoveContainer" containerID="942695f33b7f73580d1537a6f84474adc03798f8501d1c021f038d3add2c8eee"
Aug 01 00:10:52.485624 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:52.485600    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"]
Aug 01 00:10:52.488194 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:52.488174    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-5th7f"]
Aug 01 00:10:52.490163 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:52.490147    2546 scope.go:117] "RemoveContainer" containerID="9548c7f11add71a524d78e12c5c46f4804e8e4697113c531da391eadf90ab3d9"
Aug 01 00:10:52.490536 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:10:52.490507    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9548c7f11add71a524d78e12c5c46f4804e8e4697113c531da391eadf90ab3d9\": container with ID starting with 9548c7f11add71a524d78e12c5c46f4804e8e4697113c531da391eadf90ab3d9 not found: ID does not exist" containerID="9548c7f11add71a524d78e12c5c46f4804e8e4697113c531da391eadf90ab3d9"
Aug 01 00:10:52.490597 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:52.490537    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9548c7f11add71a524d78e12c5c46f4804e8e4697113c531da391eadf90ab3d9"} err="failed to get container status \"9548c7f11add71a524d78e12c5c46f4804e8e4697113c531da391eadf90ab3d9\": rpc error: code = NotFound desc = could not find container \"9548c7f11add71a524d78e12c5c46f4804e8e4697113c531da391eadf90ab3d9\": container with ID starting with 9548c7f11add71a524d78e12c5c46f4804e8e4697113c531da391eadf90ab3d9 not found: ID does not exist"
Aug 01 00:10:52.490597 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:52.490554    2546 scope.go:117] "RemoveContainer" containerID="3ec9812c28059c24a6afa524a5757c9cdd84adcafb86407c4f8dbe1c62fb992c"
Aug 01 00:10:52.490796 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:10:52.490779    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ec9812c28059c24a6afa524a5757c9cdd84adcafb86407c4f8dbe1c62fb992c\": container with ID starting with 3ec9812c28059c24a6afa524a5757c9cdd84adcafb86407c4f8dbe1c62fb992c not found: ID does not exist" containerID="3ec9812c28059c24a6afa524a5757c9cdd84adcafb86407c4f8dbe1c62fb992c"
Aug 01 00:10:52.490843 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:52.490799    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ec9812c28059c24a6afa524a5757c9cdd84adcafb86407c4f8dbe1c62fb992c"} err="failed to get container status \"3ec9812c28059c24a6afa524a5757c9cdd84adcafb86407c4f8dbe1c62fb992c\": rpc error: code = NotFound desc = could not find container \"3ec9812c28059c24a6afa524a5757c9cdd84adcafb86407c4f8dbe1c62fb992c\": container with ID starting with 3ec9812c28059c24a6afa524a5757c9cdd84adcafb86407c4f8dbe1c62fb992c not found: ID does not exist"
Aug 01 00:10:52.490843 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:52.490811    2546 scope.go:117] "RemoveContainer" containerID="942695f33b7f73580d1537a6f84474adc03798f8501d1c021f038d3add2c8eee"
Aug 01 00:10:52.491032 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:10:52.491018    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"942695f33b7f73580d1537a6f84474adc03798f8501d1c021f038d3add2c8eee\": container with ID starting with 942695f33b7f73580d1537a6f84474adc03798f8501d1c021f038d3add2c8eee not found: ID does not exist" containerID="942695f33b7f73580d1537a6f84474adc03798f8501d1c021f038d3add2c8eee"
Aug 01 00:10:52.491072 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:52.491035    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"942695f33b7f73580d1537a6f84474adc03798f8501d1c021f038d3add2c8eee"} err="failed to get container status \"942695f33b7f73580d1537a6f84474adc03798f8501d1c021f038d3add2c8eee\": rpc error: code = NotFound desc = could not find container \"942695f33b7f73580d1537a6f84474adc03798f8501d1c021f038d3add2c8eee\": container with ID starting with 942695f33b7f73580d1537a6f84474adc03798f8501d1c021f038d3add2c8eee not found: ID does not exist"
Aug 01 00:10:54.075785 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:54.075752    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" path="/var/lib/kubelet/pods/64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b/volumes"
Aug 01 00:10:55.457872 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:55.457842    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:10:55.458372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:55.458344    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:10:55.461336 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:55.461312    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:10:55.747116 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:10:55.747028    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:05.458270 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:11:05.458218    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:11:15.458463 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:11:15.458414    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:11:21.313506 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:11:21.313473    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:11:25.458566 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:11:25.458523    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:11:35.458709 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:11:35.458665    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:11:45.458676 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:11:45.458630    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:11:55.459086 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:11:55.459040    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:12:04.075500 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:04.075473    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:12:12.906440 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:12.906406    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"]
Aug 01 00:12:12.906915 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:12.906702    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kserve-container" containerID="cri-o://66fa1f0171c35bdee7214f4b08bf6b72e5081839b77adba13351209921c79c32" gracePeriod=30
Aug 01 00:12:12.907113 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:12.907038    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kube-rbac-proxy" containerID="cri-o://62581100141f1c59a51deea771c0d9564d986add1a1e2f120a587644b29da8ba" gracePeriod=30
Aug 01 00:12:12.927572 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:12.927538    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:13.795355 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:13.795318    2546 generic.go:356] "Generic (PLEG): container finished" podID="a24215b1-896e-4628-8359-101e45eb32d3" containerID="62581100141f1c59a51deea771c0d9564d986add1a1e2f120a587644b29da8ba" exitCode=2
Aug 01 00:12:13.795529 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:13.795385    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" event={"ID":"a24215b1-896e-4628-8359-101e45eb32d3","Type":"ContainerDied","Data":"62581100141f1c59a51deea771c0d9564d986add1a1e2f120a587644b29da8ba"}
Aug 01 00:12:14.071231 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:14.071135    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Aug 01 00:12:15.453781 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:15.453742    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Aug 01 00:12:17.651844 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.651814    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:12:17.756982 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.756890    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a24215b1-896e-4628-8359-101e45eb32d3-kserve-provision-location\") pod \"a24215b1-896e-4628-8359-101e45eb32d3\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") "
Aug 01 00:12:17.756982 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.756937    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a24215b1-896e-4628-8359-101e45eb32d3-cabundle-cert\") pod \"a24215b1-896e-4628-8359-101e45eb32d3\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") "
Aug 01 00:12:17.756982 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.756967    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a24215b1-896e-4628-8359-101e45eb32d3-proxy-tls\") pod \"a24215b1-896e-4628-8359-101e45eb32d3\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") "
Aug 01 00:12:17.757302 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.757020    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a24215b1-896e-4628-8359-101e45eb32d3-kube-rbac-proxy-sar-config\") pod \"a24215b1-896e-4628-8359-101e45eb32d3\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") "
Aug 01 00:12:17.757302 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.757045    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqt7f\" (UniqueName: \"kubernetes.io/projected/a24215b1-896e-4628-8359-101e45eb32d3-kube-api-access-hqt7f\") pod \"a24215b1-896e-4628-8359-101e45eb32d3\" (UID: \"a24215b1-896e-4628-8359-101e45eb32d3\") "
Aug 01 00:12:17.757302 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.757245    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a24215b1-896e-4628-8359-101e45eb32d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a24215b1-896e-4628-8359-101e45eb32d3" (UID: "a24215b1-896e-4628-8359-101e45eb32d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:12:17.757427 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.757397    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a24215b1-896e-4628-8359-101e45eb32d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a24215b1-896e-4628-8359-101e45eb32d3" (UID: "a24215b1-896e-4628-8359-101e45eb32d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:12:17.757462 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.757400    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a24215b1-896e-4628-8359-101e45eb32d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a24215b1-896e-4628-8359-101e45eb32d3" (UID: "a24215b1-896e-4628-8359-101e45eb32d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:12:17.759041 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.759020    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a24215b1-896e-4628-8359-101e45eb32d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a24215b1-896e-4628-8359-101e45eb32d3" (UID: "a24215b1-896e-4628-8359-101e45eb32d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:12:17.759143 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.759126    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24215b1-896e-4628-8359-101e45eb32d3-kube-api-access-hqt7f" (OuterVolumeSpecName: "kube-api-access-hqt7f") pod "a24215b1-896e-4628-8359-101e45eb32d3" (UID: "a24215b1-896e-4628-8359-101e45eb32d3"). InnerVolumeSpecName "kube-api-access-hqt7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:12:17.812060 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.812026    2546 generic.go:356] "Generic (PLEG): container finished" podID="a24215b1-896e-4628-8359-101e45eb32d3" containerID="66fa1f0171c35bdee7214f4b08bf6b72e5081839b77adba13351209921c79c32" exitCode=0
Aug 01 00:12:17.812186 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.812085    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"
Aug 01 00:12:17.812186 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.812080    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" event={"ID":"a24215b1-896e-4628-8359-101e45eb32d3","Type":"ContainerDied","Data":"66fa1f0171c35bdee7214f4b08bf6b72e5081839b77adba13351209921c79c32"}
Aug 01 00:12:17.812301 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.812215    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb" event={"ID":"a24215b1-896e-4628-8359-101e45eb32d3","Type":"ContainerDied","Data":"a13a1e68bf3da15c308127ecfcf88db4ea4da078b3687b0b0d19f70003ca1ea9"}
Aug 01 00:12:17.812301 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.812243    2546 scope.go:117] "RemoveContainer" containerID="62581100141f1c59a51deea771c0d9564d986add1a1e2f120a587644b29da8ba"
Aug 01 00:12:17.820654 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.820631    2546 scope.go:117] "RemoveContainer" containerID="66fa1f0171c35bdee7214f4b08bf6b72e5081839b77adba13351209921c79c32"
Aug 01 00:12:17.827944 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.827927    2546 scope.go:117] "RemoveContainer" containerID="6d0ce798d4f2210a0b72da4345768c7a2825a2d401770987f330c05a1cf32a36"
Aug 01 00:12:17.834744 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.834721    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"]
Aug 01 00:12:17.835832 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.835816    2546 scope.go:117] "RemoveContainer" containerID="62581100141f1c59a51deea771c0d9564d986add1a1e2f120a587644b29da8ba"
Aug 01 00:12:17.836090 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:12:17.836071    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62581100141f1c59a51deea771c0d9564d986add1a1e2f120a587644b29da8ba\": container with ID starting with 62581100141f1c59a51deea771c0d9564d986add1a1e2f120a587644b29da8ba not found: ID does not exist" containerID="62581100141f1c59a51deea771c0d9564d986add1a1e2f120a587644b29da8ba"
Aug 01 00:12:17.836152 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.836097    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62581100141f1c59a51deea771c0d9564d986add1a1e2f120a587644b29da8ba"} err="failed to get container status \"62581100141f1c59a51deea771c0d9564d986add1a1e2f120a587644b29da8ba\": rpc error: code = NotFound desc = could not find container \"62581100141f1c59a51deea771c0d9564d986add1a1e2f120a587644b29da8ba\": container with ID starting with 62581100141f1c59a51deea771c0d9564d986add1a1e2f120a587644b29da8ba not found: ID does not exist"
Aug 01 00:12:17.836152 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.836112    2546 scope.go:117] "RemoveContainer" containerID="66fa1f0171c35bdee7214f4b08bf6b72e5081839b77adba13351209921c79c32"
Aug 01 00:12:17.836365 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:12:17.836346    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66fa1f0171c35bdee7214f4b08bf6b72e5081839b77adba13351209921c79c32\": container with ID starting with 66fa1f0171c35bdee7214f4b08bf6b72e5081839b77adba13351209921c79c32 not found: ID does not exist" containerID="66fa1f0171c35bdee7214f4b08bf6b72e5081839b77adba13351209921c79c32"
Aug 01 00:12:17.836411 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.836371    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66fa1f0171c35bdee7214f4b08bf6b72e5081839b77adba13351209921c79c32"} err="failed to get container status \"66fa1f0171c35bdee7214f4b08bf6b72e5081839b77adba13351209921c79c32\": rpc error: code = NotFound desc = could not find container \"66fa1f0171c35bdee7214f4b08bf6b72e5081839b77adba13351209921c79c32\": container with ID starting with 66fa1f0171c35bdee7214f4b08bf6b72e5081839b77adba13351209921c79c32 not found: ID does not exist"
Aug 01 00:12:17.836411 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.836385    2546 scope.go:117] "RemoveContainer" containerID="6d0ce798d4f2210a0b72da4345768c7a2825a2d401770987f330c05a1cf32a36"
Aug 01 00:12:17.836605 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:12:17.836588    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0ce798d4f2210a0b72da4345768c7a2825a2d401770987f330c05a1cf32a36\": container with ID starting with 6d0ce798d4f2210a0b72da4345768c7a2825a2d401770987f330c05a1cf32a36 not found: ID does not exist" containerID="6d0ce798d4f2210a0b72da4345768c7a2825a2d401770987f330c05a1cf32a36"
Aug 01 00:12:17.836645 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.836609    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0ce798d4f2210a0b72da4345768c7a2825a2d401770987f330c05a1cf32a36"} err="failed to get container status \"6d0ce798d4f2210a0b72da4345768c7a2825a2d401770987f330c05a1cf32a36\": rpc error: code = NotFound desc = could not find container \"6d0ce798d4f2210a0b72da4345768c7a2825a2d401770987f330c05a1cf32a36\": container with ID starting with 6d0ce798d4f2210a0b72da4345768c7a2825a2d401770987f330c05a1cf32a36 not found: ID does not exist"
Aug 01 00:12:17.839297 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.839278    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-868cb56654-2dpgb"]
Aug 01 00:12:17.858045 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.858024    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a24215b1-896e-4628-8359-101e45eb32d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:12:17.858045 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.858046    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqt7f\" (UniqueName: \"kubernetes.io/projected/a24215b1-896e-4628-8359-101e45eb32d3-kube-api-access-hqt7f\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:12:17.858165 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.858056    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a24215b1-896e-4628-8359-101e45eb32d3-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:12:17.858165 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.858065    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a24215b1-896e-4628-8359-101e45eb32d3-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:12:17.858165 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:17.858075    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a24215b1-896e-4628-8359-101e45eb32d3-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:12:18.076215 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.076144    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a24215b1-896e-4628-8359-101e45eb32d3" path="/var/lib/kubelet/pods/a24215b1-896e-4628-8359-101e45eb32d3/volumes"
Aug 01 00:12:18.097071 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.097039    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:18.099679 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.099644    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"]
Aug 01 00:12:18.100037 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100021    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kube-rbac-proxy"
Aug 01 00:12:18.100120 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100039    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kube-rbac-proxy"
Aug 01 00:12:18.100120 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100063    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" containerName="storage-initializer"
Aug 01 00:12:18.100120 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100073    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" containerName="storage-initializer"
Aug 01 00:12:18.100120 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100083    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" containerName="kserve-container"
Aug 01 00:12:18.100120 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100093    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" containerName="kserve-container"
Aug 01 00:12:18.100120 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100103    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" containerName="kube-rbac-proxy"
Aug 01 00:12:18.100120 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100111    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" containerName="kube-rbac-proxy"
Aug 01 00:12:18.100120 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100121    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="storage-initializer"
Aug 01 00:12:18.100884 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100130    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="storage-initializer"
Aug 01 00:12:18.100884 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100153    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kserve-container"
Aug 01 00:12:18.100884 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100162    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kserve-container"
Aug 01 00:12:18.100884 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100241    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" containerName="kube-rbac-proxy"
Aug 01 00:12:18.100884 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100279    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kserve-container"
Aug 01 00:12:18.100884 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100293    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="64c1ca2d-08b5-4fce-92b2-f3f1d3b5983b" containerName="kserve-container"
Aug 01 00:12:18.100884 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.100303    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a24215b1-896e-4628-8359-101e45eb32d3" containerName="kube-rbac-proxy"
Aug 01 00:12:18.103926 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.103906    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.106490 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.106456    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Aug 01 00:12:18.106619 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.106457    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:12:18.106619 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.106533    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:12:18.106722 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.106632    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Aug 01 00:12:18.106722 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.106680    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:12:18.106722 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.106711    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:12:18.106871 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.106799    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:12:18.115945 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.115917    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"]
Aug 01 00:12:18.160660 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.160619    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dc8dabb-be35-4693-8523-7531f4a96cfc-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.160853 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.160665    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dc8dabb-be35-4693-8523-7531f4a96cfc-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.160853 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.160727    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjjg7\" (UniqueName: \"kubernetes.io/projected/6dc8dabb-be35-4693-8523-7531f4a96cfc-kube-api-access-pjjg7\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.160853 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.160841    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dc8dabb-be35-4693-8523-7531f4a96cfc-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.161016 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.160892    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dc8dabb-be35-4693-8523-7531f4a96cfc-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.262037 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.261992    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dc8dabb-be35-4693-8523-7531f4a96cfc-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.262037 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.262045    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dc8dabb-be35-4693-8523-7531f4a96cfc-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.262333 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.262064    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjjg7\" (UniqueName: \"kubernetes.io/projected/6dc8dabb-be35-4693-8523-7531f4a96cfc-kube-api-access-pjjg7\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.262333 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.262115    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dc8dabb-be35-4693-8523-7531f4a96cfc-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.262333 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.262141    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dc8dabb-be35-4693-8523-7531f4a96cfc-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.262333 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:12:18.262160    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Aug 01 00:12:18.262333 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:12:18.262248    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dc8dabb-be35-4693-8523-7531f4a96cfc-proxy-tls podName:6dc8dabb-be35-4693-8523-7531f4a96cfc nodeName:}" failed. No retries permitted until 2026-08-01 00:12:18.762226562 +0000 UTC m=+1569.221689354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6dc8dabb-be35-4693-8523-7531f4a96cfc-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4" (UID: "6dc8dabb-be35-4693-8523-7531f4a96cfc") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Aug 01 00:12:18.262631 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.262578    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dc8dabb-be35-4693-8523-7531f4a96cfc-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.262764 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.262740    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dc8dabb-be35-4693-8523-7531f4a96cfc-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.262879 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.262860    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dc8dabb-be35-4693-8523-7531f4a96cfc-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.270134 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.270109    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjjg7\" (UniqueName: \"kubernetes.io/projected/6dc8dabb-be35-4693-8523-7531f4a96cfc-kube-api-access-pjjg7\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.767059 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.767022    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dc8dabb-be35-4693-8523-7531f4a96cfc-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:18.769417 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:18.769393    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dc8dabb-be35-4693-8523-7531f4a96cfc-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:19.014172 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:19.014129    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:12:19.147372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:19.147268    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"]
Aug 01 00:12:19.150019 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:12:19.149993    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc8dabb_be35_4693_8523_7531f4a96cfc.slice/crio-3ba4b6222f4fca876a2c76052623e60581455721bad156b8137b7db8a26ead60 WatchSource:0}: Error finding container 3ba4b6222f4fca876a2c76052623e60581455721bad156b8137b7db8a26ead60: Status 404 returned error can't find the container with id 3ba4b6222f4fca876a2c76052623e60581455721bad156b8137b7db8a26ead60
Aug 01 00:12:19.555729 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:19.555694    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:12:19.821510 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:19.821419    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4" event={"ID":"6dc8dabb-be35-4693-8523-7531f4a96cfc","Type":"ContainerStarted","Data":"3ef785918d1a32e71eb4ae6489e66fe2611a8c810fb5c54c314b8cbffffc4ec8"}
Aug 01 00:12:19.821510 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:19.821498    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4" event={"ID":"6dc8dabb-be35-4693-8523-7531f4a96cfc","Type":"ContainerStarted","Data":"3ba4b6222f4fca876a2c76052623e60581455721bad156b8137b7db8a26ead60"}
Aug 01 00:12:22.834132 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:22.834101    2546 generic.go:356] "Generic (PLEG): container finished" podID="6dc8dabb-be35-4693-8523-7531f4a96cfc" containerID="3ef785918d1a32e71eb4ae6489e66fe2611a8c810fb5c54c314b8cbffffc4ec8" exitCode=0
Aug 01 00:12:22.834481 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:22.834147    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4" event={"ID":"6dc8dabb-be35-4693-8523-7531f4a96cfc","Type":"ContainerDied","Data":"3ef785918d1a32e71eb4ae6489e66fe2611a8c810fb5c54c314b8cbffffc4ec8"}
Aug 01 00:12:22.835205 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:12:22.835191    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:13:11.705529 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:11.705408    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:16.846379 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:16.846319    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:16.881641 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:16.881611    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"]
Aug 01 00:13:16.885100 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:16.885084    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:16.887883 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:16.887859    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Aug 01 00:13:16.888004 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:16.887919    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Aug 01 00:13:16.895382 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:16.895361    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"]
Aug 01 00:13:16.984822 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:16.984794    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ef25095-6bb7-41e2-861a-43bfc0595a77-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:16.984960 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:16.984850    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ef25095-6bb7-41e2-861a-43bfc0595a77-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:16.984960 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:16.984913    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ef25095-6bb7-41e2-861a-43bfc0595a77-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:16.985040 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:16.984991    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwp7c\" (UniqueName: \"kubernetes.io/projected/0ef25095-6bb7-41e2-861a-43bfc0595a77-kube-api-access-pwp7c\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:16.985086 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:16.985034    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ef25095-6bb7-41e2-861a-43bfc0595a77-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:17.085498 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:17.085475    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ef25095-6bb7-41e2-861a-43bfc0595a77-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:17.085722 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:17.085509    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ef25095-6bb7-41e2-861a-43bfc0595a77-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:17.085722 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:17.085533    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ef25095-6bb7-41e2-861a-43bfc0595a77-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:17.085722 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:17.085556    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwp7c\" (UniqueName: \"kubernetes.io/projected/0ef25095-6bb7-41e2-861a-43bfc0595a77-kube-api-access-pwp7c\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:17.085722 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:17.085576    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ef25095-6bb7-41e2-861a-43bfc0595a77-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:17.085722 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:13:17.085633    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Aug 01 00:13:17.085722 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:13:17.085695    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef25095-6bb7-41e2-861a-43bfc0595a77-proxy-tls podName:0ef25095-6bb7-41e2-861a-43bfc0595a77 nodeName:}" failed. No retries permitted until 2026-08-01 00:13:17.585676525 +0000 UTC m=+1628.045139316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0ef25095-6bb7-41e2-861a-43bfc0595a77-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" (UID: "0ef25095-6bb7-41e2-861a-43bfc0595a77") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Aug 01 00:13:17.086033 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:17.085933    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ef25095-6bb7-41e2-861a-43bfc0595a77-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:17.086192 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:17.086175    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ef25095-6bb7-41e2-861a-43bfc0595a77-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:17.086229 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:17.086213    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ef25095-6bb7-41e2-861a-43bfc0595a77-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:17.097901 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:17.097844    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwp7c\" (UniqueName: \"kubernetes.io/projected/0ef25095-6bb7-41e2-861a-43bfc0595a77-kube-api-access-pwp7c\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:17.240036 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:17.240002    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:13:17.589120 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:17.589085    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ef25095-6bb7-41e2-861a-43bfc0595a77-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:17.591415 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:17.591387    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ef25095-6bb7-41e2-861a-43bfc0595a77-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:17.796939 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:17.796887    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:17.950440 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:17.950407    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"]
Aug 01 00:13:17.954136 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:13:17.954109    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef25095_6bb7_41e2_861a_43bfc0595a77.slice/crio-00ab919ad33065f6c322536af4f3c0ff656255e7bcfade6eed306aab547fada5 WatchSource:0}: Error finding container 00ab919ad33065f6c322536af4f3c0ff656255e7bcfade6eed306aab547fada5: Status 404 returned error can't find the container with id 00ab919ad33065f6c322536af4f3c0ff656255e7bcfade6eed306aab547fada5
Aug 01 00:13:18.095688 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:18.095654    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" event={"ID":"0ef25095-6bb7-41e2-861a-43bfc0595a77","Type":"ContainerStarted","Data":"1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4"}
Aug 01 00:13:18.095875 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:18.095718    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" event={"ID":"0ef25095-6bb7-41e2-861a-43bfc0595a77","Type":"ContainerStarted","Data":"00ab919ad33065f6c322536af4f3c0ff656255e7bcfade6eed306aab547fada5"}
Aug 01 00:13:26.132833 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:26.132801    2546 generic.go:356] "Generic (PLEG): container finished" podID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerID="1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4" exitCode=0
Aug 01 00:13:26.133275 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:26.132836    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" event={"ID":"0ef25095-6bb7-41e2-861a-43bfc0595a77","Type":"ContainerDied","Data":"1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4"}
Aug 01 00:13:27.140465 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:27.140418    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" event={"ID":"0ef25095-6bb7-41e2-861a-43bfc0595a77","Type":"ContainerStarted","Data":"cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1"}
Aug 01 00:13:27.140950 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:27.140495    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" event={"ID":"0ef25095-6bb7-41e2-861a-43bfc0595a77","Type":"ContainerStarted","Data":"e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90"}
Aug 01 00:13:27.140950 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:27.140940    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:27.141100 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:27.141009    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:27.142898 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:27.142867    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Aug 01 00:13:27.160096 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:27.160050    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podStartSLOduration=3.3335575029999998 podStartE2EDuration="11.160030325s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.826472822s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:13:27.158412031 +0000 UTC m=+1637.617874847" watchObservedRunningTime="2026-08-01 00:13:27.160030325 +0000 UTC m=+1637.619493139"
Aug 01 00:13:28.145234 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:28.145188    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Aug 01 00:13:33.150565 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:33.150532    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:13:33.151074 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:33.151039    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Aug 01 00:13:43.152203 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:13:43.152174    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:14:00.599153 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:00.599123    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"]
Aug 01 00:14:00.599714 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:00.599426    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kserve-container" containerID="cri-o://e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90" gracePeriod=30
Aug 01 00:14:00.599714 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:00.599466    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kube-rbac-proxy" containerID="cri-o://cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1" gracePeriod=30
Aug 01 00:14:00.613718 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:00.613688    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:03.145852 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:03.145804    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Aug 01 00:14:03.151185 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:03.151149    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Aug 01 00:14:05.745725 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:05.745686    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:06.057070 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:06.056948    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:08.146133 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:08.146088    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Aug 01 00:14:13.146097 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:13.146050    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Aug 01 00:14:13.146637 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:13.146222    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:14:13.152182 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:13.152141    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Aug 01 00:14:18.145648 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:18.145600    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Aug 01 00:14:23.146394 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:23.146346    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Aug 01 00:14:23.151775 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:23.151737    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Aug 01 00:14:23.151926 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:23.151881    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:14:23.869061 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:23.869018    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:28.145556 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:28.145515    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Aug 01 00:14:33.145982 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:33.145931    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Aug 01 00:14:33.151307 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:33.151276    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Aug 01 00:14:38.003676 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.003654    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:14:38.135229 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.135200    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ef25095-6bb7-41e2-861a-43bfc0595a77-proxy-tls\") pod \"0ef25095-6bb7-41e2-861a-43bfc0595a77\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") "
Aug 01 00:14:38.135229 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.135233    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwp7c\" (UniqueName: \"kubernetes.io/projected/0ef25095-6bb7-41e2-861a-43bfc0595a77-kube-api-access-pwp7c\") pod \"0ef25095-6bb7-41e2-861a-43bfc0595a77\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") "
Aug 01 00:14:38.135542 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.135298    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ef25095-6bb7-41e2-861a-43bfc0595a77-kube-rbac-proxy-sar-config\") pod \"0ef25095-6bb7-41e2-861a-43bfc0595a77\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") "
Aug 01 00:14:38.135542 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.135341    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ef25095-6bb7-41e2-861a-43bfc0595a77-cabundle-cert\") pod \"0ef25095-6bb7-41e2-861a-43bfc0595a77\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") "
Aug 01 00:14:38.135542 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.135381    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ef25095-6bb7-41e2-861a-43bfc0595a77-kserve-provision-location\") pod \"0ef25095-6bb7-41e2-861a-43bfc0595a77\" (UID: \"0ef25095-6bb7-41e2-861a-43bfc0595a77\") "
Aug 01 00:14:38.135772 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.135738    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef25095-6bb7-41e2-861a-43bfc0595a77-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0ef25095-6bb7-41e2-861a-43bfc0595a77" (UID: "0ef25095-6bb7-41e2-861a-43bfc0595a77"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:14:38.135772 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.135752    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef25095-6bb7-41e2-861a-43bfc0595a77-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0ef25095-6bb7-41e2-861a-43bfc0595a77" (UID: "0ef25095-6bb7-41e2-861a-43bfc0595a77"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:14:38.136128 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.136110    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef25095-6bb7-41e2-861a-43bfc0595a77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0ef25095-6bb7-41e2-861a-43bfc0595a77" (UID: "0ef25095-6bb7-41e2-861a-43bfc0595a77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:14:38.137846 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.137821    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef25095-6bb7-41e2-861a-43bfc0595a77-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0ef25095-6bb7-41e2-861a-43bfc0595a77" (UID: "0ef25095-6bb7-41e2-861a-43bfc0595a77"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:14:38.137953 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.137822    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef25095-6bb7-41e2-861a-43bfc0595a77-kube-api-access-pwp7c" (OuterVolumeSpecName: "kube-api-access-pwp7c") pod "0ef25095-6bb7-41e2-861a-43bfc0595a77" (UID: "0ef25095-6bb7-41e2-861a-43bfc0595a77"). InnerVolumeSpecName "kube-api-access-pwp7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:14:38.236263 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.236189    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ef25095-6bb7-41e2-861a-43bfc0595a77-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:14:38.236263 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.236217    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ef25095-6bb7-41e2-861a-43bfc0595a77-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:14:38.236263 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.236231    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ef25095-6bb7-41e2-861a-43bfc0595a77-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:14:38.236263 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.236244    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ef25095-6bb7-41e2-861a-43bfc0595a77-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:14:38.236617 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.236270    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwp7c\" (UniqueName: \"kubernetes.io/projected/0ef25095-6bb7-41e2-861a-43bfc0595a77-kube-api-access-pwp7c\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:14:38.482746 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.482708    2546 generic.go:356] "Generic (PLEG): container finished" podID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerID="cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1" exitCode=2
Aug 01 00:14:38.482746 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.482733    2546 generic.go:356] "Generic (PLEG): container finished" podID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerID="e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90" exitCode=0
Aug 01 00:14:38.482945 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.482782    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" event={"ID":"0ef25095-6bb7-41e2-861a-43bfc0595a77","Type":"ContainerDied","Data":"cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1"}
Aug 01 00:14:38.482945 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.482840    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" event={"ID":"0ef25095-6bb7-41e2-861a-43bfc0595a77","Type":"ContainerDied","Data":"e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90"}
Aug 01 00:14:38.482945 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.482860    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls" event={"ID":"0ef25095-6bb7-41e2-861a-43bfc0595a77","Type":"ContainerDied","Data":"00ab919ad33065f6c322536af4f3c0ff656255e7bcfade6eed306aab547fada5"}
Aug 01 00:14:38.482945 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.482788    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"
Aug 01 00:14:38.482945 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.482884    2546 scope.go:117] "RemoveContainer" containerID="cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1"
Aug 01 00:14:38.490754 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.490626    2546 scope.go:117] "RemoveContainer" containerID="e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90"
Aug 01 00:14:38.497884 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.497865    2546 scope.go:117] "RemoveContainer" containerID="1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4"
Aug 01 00:14:38.505674 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.505619    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"]
Aug 01 00:14:38.506066 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.506047    2546 scope.go:117] "RemoveContainer" containerID="cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1"
Aug 01 00:14:38.506338 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:14:38.506320    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1\": container with ID starting with cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1 not found: ID does not exist" containerID="cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1"
Aug 01 00:14:38.506394 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.506344    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1"} err="failed to get container status \"cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1\": rpc error: code = NotFound desc = could not find container \"cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1\": container with ID starting with cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1 not found: ID does not exist"
Aug 01 00:14:38.506394 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.506359    2546 scope.go:117] "RemoveContainer" containerID="e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90"
Aug 01 00:14:38.506587 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:14:38.506571    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90\": container with ID starting with e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90 not found: ID does not exist" containerID="e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90"
Aug 01 00:14:38.506628 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.506591    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90"} err="failed to get container status \"e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90\": rpc error: code = NotFound desc = could not find container \"e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90\": container with ID starting with e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90 not found: ID does not exist"
Aug 01 00:14:38.506628 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.506605    2546 scope.go:117] "RemoveContainer" containerID="1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4"
Aug 01 00:14:38.506812 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:14:38.506795    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4\": container with ID starting with 1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4 not found: ID does not exist" containerID="1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4"
Aug 01 00:14:38.506854 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.506816    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4"} err="failed to get container status \"1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4\": rpc error: code = NotFound desc = could not find container \"1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4\": container with ID starting with 1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4 not found: ID does not exist"
Aug 01 00:14:38.506854 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.506827    2546 scope.go:117] "RemoveContainer" containerID="cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1"
Aug 01 00:14:38.507040 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.507017    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1"} err="failed to get container status \"cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1\": rpc error: code = NotFound desc = could not find container \"cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1\": container with ID starting with cdc68d81ee014c2d201ff6a1d5d1fbeed4217c237650f270cccee58ec03747a1 not found: ID does not exist"
Aug 01 00:14:38.507092 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.507043    2546 scope.go:117] "RemoveContainer" containerID="e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90"
Aug 01 00:14:38.507235 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.507217    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90"} err="failed to get container status \"e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90\": rpc error: code = NotFound desc = could not find container \"e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90\": container with ID starting with e71020e98d566c1cd31590e1f1b64796474c78cbaca5fb21dd0ccb78dd649d90 not found: ID does not exist"
Aug 01 00:14:38.507296 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.507235    2546 scope.go:117] "RemoveContainer" containerID="1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4"
Aug 01 00:14:38.507478 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.507456    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4"} err="failed to get container status \"1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4\": rpc error: code = NotFound desc = could not find container \"1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4\": container with ID starting with 1d67d83177150b7a1230c62a8aad5ba27ac8f070a020563d77e7a840b43773b4 not found: ID does not exist"
Aug 01 00:14:38.507799 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:38.507783    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-m7fls"]
Aug 01 00:14:39.490676 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:39.490638    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4" event={"ID":"6dc8dabb-be35-4693-8523-7531f4a96cfc","Type":"ContainerStarted","Data":"4b34d33f18fedf8c9ebdbae886e0ea7a25eaf27185050f99754c5a04283376a5"}
Aug 01 00:14:39.491077 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:39.490703    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4" event={"ID":"6dc8dabb-be35-4693-8523-7531f4a96cfc","Type":"ContainerStarted","Data":"ea44170ed26fbc731b2809f3d2d35cf4e6cc0c6483fae7d8ea054d952b34a411"}
Aug 01 00:14:39.491077 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:39.490802    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:14:39.510740 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:39.510694    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4" podStartSLOduration=2.350966626 podStartE2EDuration="2m21.510681479s" totalImagesPullingTime="2m15.55785031s" totalInitContainerRuntime="3.601864543s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:14:39.509210458 +0000 UTC m=+1709.968673271" watchObservedRunningTime="2026-08-01 00:14:39.510681479 +0000 UTC m=+1709.970144293"
Aug 01 00:14:40.077201 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:40.077162    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" path="/var/lib/kubelet/pods/0ef25095-6bb7-41e2-861a-43bfc0595a77/volumes"
Aug 01 00:14:40.494709 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:40.494679    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:14:41.004744 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:41.004717    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:46.137861 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.137831    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:46.171434 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.171401    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"]
Aug 01 00:14:46.171755 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.171742    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kube-rbac-proxy"
Aug 01 00:14:46.171800 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.171757    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kube-rbac-proxy"
Aug 01 00:14:46.171800 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.171771    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kserve-container"
Aug 01 00:14:46.171800 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.171776    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kserve-container"
Aug 01 00:14:46.171800 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.171790    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="storage-initializer"
Aug 01 00:14:46.171800 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.171796    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="storage-initializer"
Aug 01 00:14:46.172002 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.171850    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kube-rbac-proxy"
Aug 01 00:14:46.172002 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.171859    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ef25095-6bb7-41e2-861a-43bfc0595a77" containerName="kserve-container"
Aug 01 00:14:46.208647 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.208606    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:46.252851 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.252809    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"]
Aug 01 00:14:46.252851 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.252859    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"]
Aug 01 00:14:46.253101 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.252939    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.255896 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.255869    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Aug 01 00:14:46.256153 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.256136    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Aug 01 00:14:46.273771 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.273742    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:46.275789 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.275775    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"]
Aug 01 00:14:46.275877 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.275867    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.278358 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.278339    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Aug 01 00:14:46.404965 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.404866    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-c4c849674-7fk4s\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.404965 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.404909    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a56aabd0-e071-4551-b21b-72d6688b23dd-proxy-tls\") pod \"isvc-canary-create-predictor-8644bc947f-8rzrv\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.405173 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.404985    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd7cbdd3-f124-47d7-8e03-b9448bf14770-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-c4c849674-7fk4s\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.405173 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.405036    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a56aabd0-e071-4551-b21b-72d6688b23dd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8644bc947f-8rzrv\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.405173 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.405054    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh7gl\" (UniqueName: \"kubernetes.io/projected/a56aabd0-e071-4551-b21b-72d6688b23dd-kube-api-access-gh7gl\") pod \"isvc-canary-create-predictor-8644bc947f-8rzrv\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.405173 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.405070    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtvzg\" (UniqueName: \"kubernetes.io/projected/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kube-api-access-rtvzg\") pod \"isvc-canary-create-v2-predictor-c4c849674-7fk4s\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.405173 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.405138    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd7cbdd3-f124-47d7-8e03-b9448bf14770-proxy-tls\") pod \"isvc-canary-create-v2-predictor-c4c849674-7fk4s\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.405401 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.405182    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a56aabd0-e071-4551-b21b-72d6688b23dd-cabundle-cert\") pod \"isvc-canary-create-predictor-8644bc947f-8rzrv\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.405401 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.405200    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-c4c849674-7fk4s\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.405401 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.405245    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a56aabd0-e071-4551-b21b-72d6688b23dd-kserve-provision-location\") pod \"isvc-canary-create-predictor-8644bc947f-8rzrv\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.503263 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.503216    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:14:46.504507 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.504486    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:14:46.506129 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.506110    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd7cbdd3-f124-47d7-8e03-b9448bf14770-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-c4c849674-7fk4s\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.506191 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.506146    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a56aabd0-e071-4551-b21b-72d6688b23dd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8644bc947f-8rzrv\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.506191 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.506166    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh7gl\" (UniqueName: \"kubernetes.io/projected/a56aabd0-e071-4551-b21b-72d6688b23dd-kube-api-access-gh7gl\") pod \"isvc-canary-create-predictor-8644bc947f-8rzrv\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.506191 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.506183    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtvzg\" (UniqueName: \"kubernetes.io/projected/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kube-api-access-rtvzg\") pod \"isvc-canary-create-v2-predictor-c4c849674-7fk4s\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.506395 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.506206    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd7cbdd3-f124-47d7-8e03-b9448bf14770-proxy-tls\") pod \"isvc-canary-create-v2-predictor-c4c849674-7fk4s\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.506395 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.506241    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a56aabd0-e071-4551-b21b-72d6688b23dd-cabundle-cert\") pod \"isvc-canary-create-predictor-8644bc947f-8rzrv\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.506395 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.506285    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-c4c849674-7fk4s\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.506395 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.506319    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a56aabd0-e071-4551-b21b-72d6688b23dd-kserve-provision-location\") pod \"isvc-canary-create-predictor-8644bc947f-8rzrv\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.506395 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.506355    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-c4c849674-7fk4s\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.506650 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.506513    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a56aabd0-e071-4551-b21b-72d6688b23dd-proxy-tls\") pod \"isvc-canary-create-predictor-8644bc947f-8rzrv\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.506863 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.506836    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a56aabd0-e071-4551-b21b-72d6688b23dd-kserve-provision-location\") pod \"isvc-canary-create-predictor-8644bc947f-8rzrv\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.507001 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.506980    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a56aabd0-e071-4551-b21b-72d6688b23dd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8644bc947f-8rzrv\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.507087 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.507028    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a56aabd0-e071-4551-b21b-72d6688b23dd-cabundle-cert\") pod \"isvc-canary-create-predictor-8644bc947f-8rzrv\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.507087 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.507028    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-c4c849674-7fk4s\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.507502 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.507475    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd7cbdd3-f124-47d7-8e03-b9448bf14770-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-c4c849674-7fk4s\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.507630 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.507475    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-c4c849674-7fk4s\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.508833 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.508814    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd7cbdd3-f124-47d7-8e03-b9448bf14770-proxy-tls\") pod \"isvc-canary-create-v2-predictor-c4c849674-7fk4s\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.509014 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.508996    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a56aabd0-e071-4551-b21b-72d6688b23dd-proxy-tls\") pod \"isvc-canary-create-predictor-8644bc947f-8rzrv\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.513358 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.513333    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtvzg\" (UniqueName: \"kubernetes.io/projected/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kube-api-access-rtvzg\") pod \"isvc-canary-create-v2-predictor-c4c849674-7fk4s\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.513436 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.513392    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh7gl\" (UniqueName: \"kubernetes.io/projected/a56aabd0-e071-4551-b21b-72d6688b23dd-kube-api-access-gh7gl\") pod \"isvc-canary-create-predictor-8644bc947f-8rzrv\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.564051 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.564013    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:46.585214 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.585177    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:46.740903 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.740709    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"]
Aug 01 00:14:46.743168 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:14:46.743141    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56aabd0_e071_4551_b21b_72d6688b23dd.slice/crio-f264867bc9ecdfaa8066451235408dbd2703336a53a1f9c7f85ca3b1bd3114e4 WatchSource:0}: Error finding container f264867bc9ecdfaa8066451235408dbd2703336a53a1f9c7f85ca3b1bd3114e4: Status 404 returned error can't find the container with id f264867bc9ecdfaa8066451235408dbd2703336a53a1f9c7f85ca3b1bd3114e4
Aug 01 00:14:46.781093 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:46.781068    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"]
Aug 01 00:14:46.784325 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:14:46.784295    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7cbdd3_f124_47d7_8e03_b9448bf14770.slice/crio-d609fe5ce040cbf20e838ca76ddec6e6e4af76d37cbd0686907cf4d02a51483a WatchSource:0}: Error finding container d609fe5ce040cbf20e838ca76ddec6e6e4af76d37cbd0686907cf4d02a51483a: Status 404 returned error can't find the container with id d609fe5ce040cbf20e838ca76ddec6e6e4af76d37cbd0686907cf4d02a51483a
Aug 01 00:14:47.526964 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:47.526918    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" event={"ID":"cd7cbdd3-f124-47d7-8e03-b9448bf14770","Type":"ContainerStarted","Data":"48b9731584bd42c5bbfe09849d8add9e94714a8019628d4aeb97513d9a57c59d"}
Aug 01 00:14:47.527558 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:47.526982    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" event={"ID":"cd7cbdd3-f124-47d7-8e03-b9448bf14770","Type":"ContainerStarted","Data":"d609fe5ce040cbf20e838ca76ddec6e6e4af76d37cbd0686907cf4d02a51483a"}
Aug 01 00:14:47.529292 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:47.529243    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" event={"ID":"a56aabd0-e071-4551-b21b-72d6688b23dd","Type":"ContainerStarted","Data":"7802e2467f36a627ee671c7147a1703070adbd381f00008126a9d0979891bb0f"}
Aug 01 00:14:47.529435 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:47.529332    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" event={"ID":"a56aabd0-e071-4551-b21b-72d6688b23dd","Type":"ContainerStarted","Data":"f264867bc9ecdfaa8066451235408dbd2703336a53a1f9c7f85ca3b1bd3114e4"}
Aug 01 00:14:51.550390 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:51.550355    2546 generic.go:356] "Generic (PLEG): container finished" podID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerID="48b9731584bd42c5bbfe09849d8add9e94714a8019628d4aeb97513d9a57c59d" exitCode=0
Aug 01 00:14:51.550822 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:51.550431    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" event={"ID":"cd7cbdd3-f124-47d7-8e03-b9448bf14770","Type":"ContainerDied","Data":"48b9731584bd42c5bbfe09849d8add9e94714a8019628d4aeb97513d9a57c59d"}
Aug 01 00:14:51.551939 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:51.551917    2546 generic.go:356] "Generic (PLEG): container finished" podID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerID="7802e2467f36a627ee671c7147a1703070adbd381f00008126a9d0979891bb0f" exitCode=0
Aug 01 00:14:51.552013 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:51.551962    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" event={"ID":"a56aabd0-e071-4551-b21b-72d6688b23dd","Type":"ContainerDied","Data":"7802e2467f36a627ee671c7147a1703070adbd381f00008126a9d0979891bb0f"}
Aug 01 00:14:52.564041 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:52.563998    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" event={"ID":"cd7cbdd3-f124-47d7-8e03-b9448bf14770","Type":"ContainerStarted","Data":"6a70439b0a76f457fad3f2ef32eee3b6f01ccfed0f52b51ac337705e14fd8a78"}
Aug 01 00:14:52.564518 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:52.564096    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" event={"ID":"cd7cbdd3-f124-47d7-8e03-b9448bf14770","Type":"ContainerStarted","Data":"277f1d5f0c1d4e6a0939aa476f4801d816d96ce82e7d1386701b34c14a715349"}
Aug 01 00:14:52.564518 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:52.564295    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:52.566124 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:52.566101    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" event={"ID":"a56aabd0-e071-4551-b21b-72d6688b23dd","Type":"ContainerStarted","Data":"a842cbd83969ecd02b2ce2487c6bd55789648121001fc825b99e7720024e5c02"}
Aug 01 00:14:52.566225 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:52.566148    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" event={"ID":"a56aabd0-e071-4551-b21b-72d6688b23dd","Type":"ContainerStarted","Data":"be58b35041e8265f21b82d7bc865ab59bd93f5b37a4ffdf76fed16356b5008af"}
Aug 01 00:14:52.566343 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:52.566326    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:52.584311 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:52.584267    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" podStartSLOduration=2.52419386 podStartE2EDuration="6.584241831s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.060047971s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:14:52.58298577 +0000 UTC m=+1723.042448582" watchObservedRunningTime="2026-08-01 00:14:52.584241831 +0000 UTC m=+1723.043704644"
Aug 01 00:14:52.601892 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:52.601846    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" podStartSLOduration=2.560606387 podStartE2EDuration="6.601833976s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.041227589s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:14:52.600848876 +0000 UTC m=+1723.060311689" watchObservedRunningTime="2026-08-01 00:14:52.601833976 +0000 UTC m=+1723.061296790"
Aug 01 00:14:53.571617 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:53.571581    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:53.571996 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:53.571648    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:53.572714 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:53.572684    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Aug 01 00:14:53.572810 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:53.572684    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Aug 01 00:14:54.575931 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:54.575890    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Aug 01 00:14:54.576335 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:54.576033    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Aug 01 00:14:59.580554 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:59.580515    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:14:59.581223 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:59.581191    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Aug 01 00:14:59.581360 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:59.581275    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:14:59.581782 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:14:59.581757    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Aug 01 00:15:09.581241 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:09.581200    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Aug 01 00:15:09.581738 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:09.581625    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Aug 01 00:15:16.521837 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:16.521786    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4" podUID="6dc8dabb-be35-4693-8523-7531f4a96cfc" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Aug 01 00:15:19.581515 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:19.581476    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Aug 01 00:15:19.581971 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:19.581634    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Aug 01 00:15:26.507185 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:26.507143    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:15:28.228317 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:28.228283    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"]
Aug 01 00:15:28.228794 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:28.228530    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4" podUID="6dc8dabb-be35-4693-8523-7531f4a96cfc" containerName="kserve-container" containerID="cri-o://ea44170ed26fbc731b2809f3d2d35cf4e6cc0c6483fae7d8ea054d952b34a411" gracePeriod=30
Aug 01 00:15:28.228794 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:28.228576    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4" podUID="6dc8dabb-be35-4693-8523-7531f4a96cfc" containerName="kube-rbac-proxy" containerID="cri-o://4b34d33f18fedf8c9ebdbae886e0ea7a25eaf27185050f99754c5a04283376a5" gracePeriod=30
Aug 01 00:15:28.249329 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:28.249301    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:28.731112 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:28.731077    2546 generic.go:356] "Generic (PLEG): container finished" podID="6dc8dabb-be35-4693-8523-7531f4a96cfc" containerID="4b34d33f18fedf8c9ebdbae886e0ea7a25eaf27185050f99754c5a04283376a5" exitCode=2
Aug 01 00:15:28.731281 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:28.731148    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4" event={"ID":"6dc8dabb-be35-4693-8523-7531f4a96cfc","Type":"ContainerDied","Data":"4b34d33f18fedf8c9ebdbae886e0ea7a25eaf27185050f99754c5a04283376a5"}
Aug 01 00:15:29.581376 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:29.581341    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Aug 01 00:15:29.581812 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:29.581664    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Aug 01 00:15:31.500157 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:31.500116    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4" podUID="6dc8dabb-be35-4693-8523-7531f4a96cfc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Aug 01 00:15:33.392100 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.392053    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:33.433724 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.433689    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"]
Aug 01 00:15:33.436185 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.436168    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.438554 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.438529    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Aug 01 00:15:33.438667 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.438573    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Aug 01 00:15:33.447658 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.447636    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"]
Aug 01 00:15:33.507052 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.507025    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.507191 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.507099    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-828k4\" (UniqueName: \"kubernetes.io/projected/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kube-api-access-828k4\") pod \"isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.507274 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.507187    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/649b43d3-2f38-4a73-b5fa-3f7f89b18641-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.507274 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.507214    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.507274 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.507235    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/649b43d3-2f38-4a73-b5fa-3f7f89b18641-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.608268 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.608213    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/649b43d3-2f38-4a73-b5fa-3f7f89b18641-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.608467 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.608274    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.608467 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.608304    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/649b43d3-2f38-4a73-b5fa-3f7f89b18641-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.608467 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.608342    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.608467 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.608408    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828k4\" (UniqueName: \"kubernetes.io/projected/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kube-api-access-828k4\") pod \"isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.608773 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.608746    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.609038 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.609018    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/649b43d3-2f38-4a73-b5fa-3f7f89b18641-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.609081 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.609049    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.610692 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.610670    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/649b43d3-2f38-4a73-b5fa-3f7f89b18641-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.616913 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.616894    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-828k4\" (UniqueName: \"kubernetes.io/projected/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kube-api-access-828k4\") pod \"isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.636107 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.636079    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:15:33.745892 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.745815    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:33.888997 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:33.888967    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"]
Aug 01 00:15:33.891706 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:15:33.891684    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649b43d3_2f38_4a73_b5fa_3f7f89b18641.slice/crio-cc1c14fd4578748af5401e1fdb8a7110bed24a42a56311df4a2959a134c320e5 WatchSource:0}: Error finding container cc1c14fd4578748af5401e1fdb8a7110bed24a42a56311df4a2959a134c320e5: Status 404 returned error can't find the container with id cc1c14fd4578748af5401e1fdb8a7110bed24a42a56311df4a2959a134c320e5
Aug 01 00:15:34.758484 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:34.758447    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj" event={"ID":"649b43d3-2f38-4a73-b5fa-3f7f89b18641","Type":"ContainerStarted","Data":"3f2031f99a8b2dcad3d964b7a5f1da36a126eac390bba1d18cf2286c853bc8c7"}
Aug 01 00:15:34.758946 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:34.758548    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj" event={"ID":"649b43d3-2f38-4a73-b5fa-3f7f89b18641","Type":"ContainerStarted","Data":"cc1c14fd4578748af5401e1fdb8a7110bed24a42a56311df4a2959a134c320e5"}
Aug 01 00:15:35.690411 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.690386    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:15:35.765615 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.765583    2546 generic.go:356] "Generic (PLEG): container finished" podID="6dc8dabb-be35-4693-8523-7531f4a96cfc" containerID="ea44170ed26fbc731b2809f3d2d35cf4e6cc0c6483fae7d8ea054d952b34a411" exitCode=0
Aug 01 00:15:35.765988 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.765646    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4" event={"ID":"6dc8dabb-be35-4693-8523-7531f4a96cfc","Type":"ContainerDied","Data":"ea44170ed26fbc731b2809f3d2d35cf4e6cc0c6483fae7d8ea054d952b34a411"}
Aug 01 00:15:35.765988 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.765696    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"
Aug 01 00:15:35.765988 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.765705    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4" event={"ID":"6dc8dabb-be35-4693-8523-7531f4a96cfc","Type":"ContainerDied","Data":"3ba4b6222f4fca876a2c76052623e60581455721bad156b8137b7db8a26ead60"}
Aug 01 00:15:35.765988 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.765731    2546 scope.go:117] "RemoveContainer" containerID="4b34d33f18fedf8c9ebdbae886e0ea7a25eaf27185050f99754c5a04283376a5"
Aug 01 00:15:35.773741 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.773722    2546 scope.go:117] "RemoveContainer" containerID="ea44170ed26fbc731b2809f3d2d35cf4e6cc0c6483fae7d8ea054d952b34a411"
Aug 01 00:15:35.780824 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.780804    2546 scope.go:117] "RemoveContainer" containerID="3ef785918d1a32e71eb4ae6489e66fe2611a8c810fb5c54c314b8cbffffc4ec8"
Aug 01 00:15:35.787876 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.787856    2546 scope.go:117] "RemoveContainer" containerID="4b34d33f18fedf8c9ebdbae886e0ea7a25eaf27185050f99754c5a04283376a5"
Aug 01 00:15:35.788136 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:15:35.788118    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b34d33f18fedf8c9ebdbae886e0ea7a25eaf27185050f99754c5a04283376a5\": container with ID starting with 4b34d33f18fedf8c9ebdbae886e0ea7a25eaf27185050f99754c5a04283376a5 not found: ID does not exist" containerID="4b34d33f18fedf8c9ebdbae886e0ea7a25eaf27185050f99754c5a04283376a5"
Aug 01 00:15:35.788183 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.788144    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b34d33f18fedf8c9ebdbae886e0ea7a25eaf27185050f99754c5a04283376a5"} err="failed to get container status \"4b34d33f18fedf8c9ebdbae886e0ea7a25eaf27185050f99754c5a04283376a5\": rpc error: code = NotFound desc = could not find container \"4b34d33f18fedf8c9ebdbae886e0ea7a25eaf27185050f99754c5a04283376a5\": container with ID starting with 4b34d33f18fedf8c9ebdbae886e0ea7a25eaf27185050f99754c5a04283376a5 not found: ID does not exist"
Aug 01 00:15:35.788183 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.788162    2546 scope.go:117] "RemoveContainer" containerID="ea44170ed26fbc731b2809f3d2d35cf4e6cc0c6483fae7d8ea054d952b34a411"
Aug 01 00:15:35.788415 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:15:35.788397    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea44170ed26fbc731b2809f3d2d35cf4e6cc0c6483fae7d8ea054d952b34a411\": container with ID starting with ea44170ed26fbc731b2809f3d2d35cf4e6cc0c6483fae7d8ea054d952b34a411 not found: ID does not exist" containerID="ea44170ed26fbc731b2809f3d2d35cf4e6cc0c6483fae7d8ea054d952b34a411"
Aug 01 00:15:35.788461 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.788421    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea44170ed26fbc731b2809f3d2d35cf4e6cc0c6483fae7d8ea054d952b34a411"} err="failed to get container status \"ea44170ed26fbc731b2809f3d2d35cf4e6cc0c6483fae7d8ea054d952b34a411\": rpc error: code = NotFound desc = could not find container \"ea44170ed26fbc731b2809f3d2d35cf4e6cc0c6483fae7d8ea054d952b34a411\": container with ID starting with ea44170ed26fbc731b2809f3d2d35cf4e6cc0c6483fae7d8ea054d952b34a411 not found: ID does not exist"
Aug 01 00:15:35.788461 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.788437    2546 scope.go:117] "RemoveContainer" containerID="3ef785918d1a32e71eb4ae6489e66fe2611a8c810fb5c54c314b8cbffffc4ec8"
Aug 01 00:15:35.788659 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:15:35.788641    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef785918d1a32e71eb4ae6489e66fe2611a8c810fb5c54c314b8cbffffc4ec8\": container with ID starting with 3ef785918d1a32e71eb4ae6489e66fe2611a8c810fb5c54c314b8cbffffc4ec8 not found: ID does not exist" containerID="3ef785918d1a32e71eb4ae6489e66fe2611a8c810fb5c54c314b8cbffffc4ec8"
Aug 01 00:15:35.788700 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.788663    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef785918d1a32e71eb4ae6489e66fe2611a8c810fb5c54c314b8cbffffc4ec8"} err="failed to get container status \"3ef785918d1a32e71eb4ae6489e66fe2611a8c810fb5c54c314b8cbffffc4ec8\": rpc error: code = NotFound desc = could not find container \"3ef785918d1a32e71eb4ae6489e66fe2611a8c810fb5c54c314b8cbffffc4ec8\": container with ID starting with 3ef785918d1a32e71eb4ae6489e66fe2611a8c810fb5c54c314b8cbffffc4ec8 not found: ID does not exist"
Aug 01 00:15:35.829070 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.829034    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dc8dabb-be35-4693-8523-7531f4a96cfc-cabundle-cert\") pod \"6dc8dabb-be35-4693-8523-7531f4a96cfc\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") "
Aug 01 00:15:35.829272 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.829149    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dc8dabb-be35-4693-8523-7531f4a96cfc-proxy-tls\") pod \"6dc8dabb-be35-4693-8523-7531f4a96cfc\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") "
Aug 01 00:15:35.829272 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.829192    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjjg7\" (UniqueName: \"kubernetes.io/projected/6dc8dabb-be35-4693-8523-7531f4a96cfc-kube-api-access-pjjg7\") pod \"6dc8dabb-be35-4693-8523-7531f4a96cfc\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") "
Aug 01 00:15:35.829272 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.829221    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dc8dabb-be35-4693-8523-7531f4a96cfc-kserve-provision-location\") pod \"6dc8dabb-be35-4693-8523-7531f4a96cfc\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") "
Aug 01 00:15:35.829272 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.829265    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dc8dabb-be35-4693-8523-7531f4a96cfc-kube-rbac-proxy-sar-config\") pod \"6dc8dabb-be35-4693-8523-7531f4a96cfc\" (UID: \"6dc8dabb-be35-4693-8523-7531f4a96cfc\") "
Aug 01 00:15:35.829536 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.829508    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc8dabb-be35-4693-8523-7531f4a96cfc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6dc8dabb-be35-4693-8523-7531f4a96cfc" (UID: "6dc8dabb-be35-4693-8523-7531f4a96cfc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:15:35.829731 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.829647    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc8dabb-be35-4693-8523-7531f4a96cfc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6dc8dabb-be35-4693-8523-7531f4a96cfc" (UID: "6dc8dabb-be35-4693-8523-7531f4a96cfc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:15:35.829839 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.829802    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc8dabb-be35-4693-8523-7531f4a96cfc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6dc8dabb-be35-4693-8523-7531f4a96cfc" (UID: "6dc8dabb-be35-4693-8523-7531f4a96cfc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:15:35.831544 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.831515    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dc8dabb-be35-4693-8523-7531f4a96cfc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6dc8dabb-be35-4693-8523-7531f4a96cfc" (UID: "6dc8dabb-be35-4693-8523-7531f4a96cfc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:15:35.831637 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.831561    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc8dabb-be35-4693-8523-7531f4a96cfc-kube-api-access-pjjg7" (OuterVolumeSpecName: "kube-api-access-pjjg7") pod "6dc8dabb-be35-4693-8523-7531f4a96cfc" (UID: "6dc8dabb-be35-4693-8523-7531f4a96cfc"). InnerVolumeSpecName "kube-api-access-pjjg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:15:35.930572 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.930537    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6dc8dabb-be35-4693-8523-7531f4a96cfc-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:15:35.930572 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.930571    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjjg7\" (UniqueName: \"kubernetes.io/projected/6dc8dabb-be35-4693-8523-7531f4a96cfc-kube-api-access-pjjg7\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:15:35.930776 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.930584    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dc8dabb-be35-4693-8523-7531f4a96cfc-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:15:35.930776 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.930598    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6dc8dabb-be35-4693-8523-7531f4a96cfc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:15:35.930776 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:35.930611    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dc8dabb-be35-4693-8523-7531f4a96cfc-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:15:36.088738 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:36.088713    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"]
Aug 01 00:15:36.092347 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:36.092326    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fx4r4"]
Aug 01 00:15:38.076096 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:38.076060    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dc8dabb-be35-4693-8523-7531f4a96cfc" path="/var/lib/kubelet/pods/6dc8dabb-be35-4693-8523-7531f4a96cfc/volumes"
Aug 01 00:15:39.581221 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:39.581125    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Aug 01 00:15:39.581786 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:39.581757    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Aug 01 00:15:39.785635 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:39.785600    2546 generic.go:356] "Generic (PLEG): container finished" podID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerID="3f2031f99a8b2dcad3d964b7a5f1da36a126eac390bba1d18cf2286c853bc8c7" exitCode=0
Aug 01 00:15:39.785838 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:39.785675    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj" event={"ID":"649b43d3-2f38-4a73-b5fa-3f7f89b18641","Type":"ContainerDied","Data":"3f2031f99a8b2dcad3d964b7a5f1da36a126eac390bba1d18cf2286c853bc8c7"}
Aug 01 00:15:40.791979 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:40.791946    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj" event={"ID":"649b43d3-2f38-4a73-b5fa-3f7f89b18641","Type":"ContainerStarted","Data":"2027e85e23c47394f4b9fc7864733727467130ba68be58db6828f77317d5722f"}
Aug 01 00:15:40.792366 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:40.792006    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj" event={"ID":"649b43d3-2f38-4a73-b5fa-3f7f89b18641","Type":"ContainerStarted","Data":"4b44bb22e4b35ee74ad862d68e9409fe29140823b8d374b08cc87c5370f0297b"}
Aug 01 00:15:40.792413 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:40.792363    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:40.792512 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:40.792491    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:40.793967 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:40.793942    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj" podUID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Aug 01 00:15:40.813118 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:40.813071    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj" podStartSLOduration=2.624011188 podStartE2EDuration="7.813055239s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.189044051s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:15:40.811500342 +0000 UTC m=+1771.270963155" watchObservedRunningTime="2026-08-01 00:15:40.813055239 +0000 UTC m=+1771.272518052"
Aug 01 00:15:41.796445 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:41.796407    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj" podUID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Aug 01 00:15:46.802914 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:46.802884    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:46.803405 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:46.803379    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj" podUID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Aug 01 00:15:49.581656 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:49.581614    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Aug 01 00:15:49.582273 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:49.581778    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Aug 01 00:15:56.804269 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:56.804224    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:15:59.581632 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:59.581587    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Aug 01 00:15:59.582044 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:15:59.581741    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Aug 01 00:16:05.447706 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:05.447669    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"]
Aug 01 00:16:05.448137 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:05.447941    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj" podUID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerName="kserve-container" containerID="cri-o://4b44bb22e4b35ee74ad862d68e9409fe29140823b8d374b08cc87c5370f0297b" gracePeriod=30
Aug 01 00:16:05.448137 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:05.447979    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj" podUID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerName="kube-rbac-proxy" containerID="cri-o://2027e85e23c47394f4b9fc7864733727467130ba68be58db6828f77317d5722f" gracePeriod=30
Aug 01 00:16:05.467859 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:05.467824    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:05.914414 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:05.914379    2546 generic.go:356] "Generic (PLEG): container finished" podID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerID="2027e85e23c47394f4b9fc7864733727467130ba68be58db6828f77317d5722f" exitCode=2
Aug 01 00:16:05.914596 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:05.914449    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj" event={"ID":"649b43d3-2f38-4a73-b5fa-3f7f89b18641","Type":"ContainerDied","Data":"2027e85e23c47394f4b9fc7864733727467130ba68be58db6828f77317d5722f"}
Aug 01 00:16:06.392566 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.392543    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:16:06.515601 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.515520    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kserve-provision-location\") pod \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") "
Aug 01 00:16:06.515601 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.515568    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-828k4\" (UniqueName: \"kubernetes.io/projected/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kube-api-access-828k4\") pod \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") "
Aug 01 00:16:06.515601 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.515602    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kube-rbac-proxy-sar-config\") pod \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") "
Aug 01 00:16:06.516131 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.515627    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/649b43d3-2f38-4a73-b5fa-3f7f89b18641-proxy-tls\") pod \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") "
Aug 01 00:16:06.516131 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.515660    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/649b43d3-2f38-4a73-b5fa-3f7f89b18641-cabundle-cert\") pod \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\" (UID: \"649b43d3-2f38-4a73-b5fa-3f7f89b18641\") "
Aug 01 00:16:06.516131 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.515976    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "649b43d3-2f38-4a73-b5fa-3f7f89b18641" (UID: "649b43d3-2f38-4a73-b5fa-3f7f89b18641"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:16:06.516131 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.516030    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/649b43d3-2f38-4a73-b5fa-3f7f89b18641-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "649b43d3-2f38-4a73-b5fa-3f7f89b18641" (UID: "649b43d3-2f38-4a73-b5fa-3f7f89b18641"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:16:06.517809 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.517784    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/649b43d3-2f38-4a73-b5fa-3f7f89b18641-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "649b43d3-2f38-4a73-b5fa-3f7f89b18641" (UID: "649b43d3-2f38-4a73-b5fa-3f7f89b18641"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:16:06.517892 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.517788    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kube-api-access-828k4" (OuterVolumeSpecName: "kube-api-access-828k4") pod "649b43d3-2f38-4a73-b5fa-3f7f89b18641" (UID: "649b43d3-2f38-4a73-b5fa-3f7f89b18641"). InnerVolumeSpecName "kube-api-access-828k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:16:06.544761 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.544725    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "649b43d3-2f38-4a73-b5fa-3f7f89b18641" (UID: "649b43d3-2f38-4a73-b5fa-3f7f89b18641"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:16:06.616683 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.616656    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-828k4\" (UniqueName: \"kubernetes.io/projected/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kube-api-access-828k4\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:16:06.616683 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.616678    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:16:06.616841 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.616690    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/649b43d3-2f38-4a73-b5fa-3f7f89b18641-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:16:06.616841 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.616700    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/649b43d3-2f38-4a73-b5fa-3f7f89b18641-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:16:06.616841 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.616709    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/649b43d3-2f38-4a73-b5fa-3f7f89b18641-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:16:06.921387 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.921352    2546 generic.go:356] "Generic (PLEG): container finished" podID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerID="4b44bb22e4b35ee74ad862d68e9409fe29140823b8d374b08cc87c5370f0297b" exitCode=0
Aug 01 00:16:06.921603 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.921410    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"
Aug 01 00:16:06.921603 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.921446    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj" event={"ID":"649b43d3-2f38-4a73-b5fa-3f7f89b18641","Type":"ContainerDied","Data":"4b44bb22e4b35ee74ad862d68e9409fe29140823b8d374b08cc87c5370f0297b"}
Aug 01 00:16:06.921603 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.921521    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj" event={"ID":"649b43d3-2f38-4a73-b5fa-3f7f89b18641","Type":"ContainerDied","Data":"cc1c14fd4578748af5401e1fdb8a7110bed24a42a56311df4a2959a134c320e5"}
Aug 01 00:16:06.921603 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.921566    2546 scope.go:117] "RemoveContainer" containerID="2027e85e23c47394f4b9fc7864733727467130ba68be58db6828f77317d5722f"
Aug 01 00:16:06.938996 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.938977    2546 scope.go:117] "RemoveContainer" containerID="4b44bb22e4b35ee74ad862d68e9409fe29140823b8d374b08cc87c5370f0297b"
Aug 01 00:16:06.946350 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.946331    2546 scope.go:117] "RemoveContainer" containerID="3f2031f99a8b2dcad3d964b7a5f1da36a126eac390bba1d18cf2286c853bc8c7"
Aug 01 00:16:06.948736 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.948714    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"]
Aug 01 00:16:06.951947 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.951925    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75b75d9b94-2dnwj"]
Aug 01 00:16:06.954487 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.954470    2546 scope.go:117] "RemoveContainer" containerID="2027e85e23c47394f4b9fc7864733727467130ba68be58db6828f77317d5722f"
Aug 01 00:16:06.954758 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:16:06.954739    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2027e85e23c47394f4b9fc7864733727467130ba68be58db6828f77317d5722f\": container with ID starting with 2027e85e23c47394f4b9fc7864733727467130ba68be58db6828f77317d5722f not found: ID does not exist" containerID="2027e85e23c47394f4b9fc7864733727467130ba68be58db6828f77317d5722f"
Aug 01 00:16:06.954825 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.954770    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2027e85e23c47394f4b9fc7864733727467130ba68be58db6828f77317d5722f"} err="failed to get container status \"2027e85e23c47394f4b9fc7864733727467130ba68be58db6828f77317d5722f\": rpc error: code = NotFound desc = could not find container \"2027e85e23c47394f4b9fc7864733727467130ba68be58db6828f77317d5722f\": container with ID starting with 2027e85e23c47394f4b9fc7864733727467130ba68be58db6828f77317d5722f not found: ID does not exist"
Aug 01 00:16:06.954825 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.954796    2546 scope.go:117] "RemoveContainer" containerID="4b44bb22e4b35ee74ad862d68e9409fe29140823b8d374b08cc87c5370f0297b"
Aug 01 00:16:06.955055 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:16:06.955036    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b44bb22e4b35ee74ad862d68e9409fe29140823b8d374b08cc87c5370f0297b\": container with ID starting with 4b44bb22e4b35ee74ad862d68e9409fe29140823b8d374b08cc87c5370f0297b not found: ID does not exist" containerID="4b44bb22e4b35ee74ad862d68e9409fe29140823b8d374b08cc87c5370f0297b"
Aug 01 00:16:06.955098 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.955062    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b44bb22e4b35ee74ad862d68e9409fe29140823b8d374b08cc87c5370f0297b"} err="failed to get container status \"4b44bb22e4b35ee74ad862d68e9409fe29140823b8d374b08cc87c5370f0297b\": rpc error: code = NotFound desc = could not find container \"4b44bb22e4b35ee74ad862d68e9409fe29140823b8d374b08cc87c5370f0297b\": container with ID starting with 4b44bb22e4b35ee74ad862d68e9409fe29140823b8d374b08cc87c5370f0297b not found: ID does not exist"
Aug 01 00:16:06.955098 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.955077    2546 scope.go:117] "RemoveContainer" containerID="3f2031f99a8b2dcad3d964b7a5f1da36a126eac390bba1d18cf2286c853bc8c7"
Aug 01 00:16:06.955327 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:16:06.955281    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2031f99a8b2dcad3d964b7a5f1da36a126eac390bba1d18cf2286c853bc8c7\": container with ID starting with 3f2031f99a8b2dcad3d964b7a5f1da36a126eac390bba1d18cf2286c853bc8c7 not found: ID does not exist" containerID="3f2031f99a8b2dcad3d964b7a5f1da36a126eac390bba1d18cf2286c853bc8c7"
Aug 01 00:16:06.955327 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:06.955309    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2031f99a8b2dcad3d964b7a5f1da36a126eac390bba1d18cf2286c853bc8c7"} err="failed to get container status \"3f2031f99a8b2dcad3d964b7a5f1da36a126eac390bba1d18cf2286c853bc8c7\": rpc error: code = NotFound desc = could not find container \"3f2031f99a8b2dcad3d964b7a5f1da36a126eac390bba1d18cf2286c853bc8c7\": container with ID starting with 3f2031f99a8b2dcad3d964b7a5f1da36a126eac390bba1d18cf2286c853bc8c7 not found: ID does not exist"
Aug 01 00:16:08.076016 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:08.075982    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" path="/var/lib/kubelet/pods/649b43d3-2f38-4a73-b5fa-3f7f89b18641/volumes"
Aug 01 00:16:09.581498 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:09.581455    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Aug 01 00:16:09.582366 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:09.582347    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:16:10.591515 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.591477    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:10.615377 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615347    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"]
Aug 01 00:16:10.615685 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615673    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerName="kube-rbac-proxy"
Aug 01 00:16:10.615728 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615686    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerName="kube-rbac-proxy"
Aug 01 00:16:10.615728 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615696    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc8dabb-be35-4693-8523-7531f4a96cfc" containerName="kube-rbac-proxy"
Aug 01 00:16:10.615728 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615701    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc8dabb-be35-4693-8523-7531f4a96cfc" containerName="kube-rbac-proxy"
Aug 01 00:16:10.615728 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615711    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerName="storage-initializer"
Aug 01 00:16:10.615728 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615717    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerName="storage-initializer"
Aug 01 00:16:10.615728 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615728    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc8dabb-be35-4693-8523-7531f4a96cfc" containerName="kserve-container"
Aug 01 00:16:10.615912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615733    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc8dabb-be35-4693-8523-7531f4a96cfc" containerName="kserve-container"
Aug 01 00:16:10.615912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615748    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerName="kserve-container"
Aug 01 00:16:10.615912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615753    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerName="kserve-container"
Aug 01 00:16:10.615912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615761    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc8dabb-be35-4693-8523-7531f4a96cfc" containerName="storage-initializer"
Aug 01 00:16:10.615912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615767    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc8dabb-be35-4693-8523-7531f4a96cfc" containerName="storage-initializer"
Aug 01 00:16:10.615912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615813    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc8dabb-be35-4693-8523-7531f4a96cfc" containerName="kserve-container"
Aug 01 00:16:10.615912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615820    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc8dabb-be35-4693-8523-7531f4a96cfc" containerName="kube-rbac-proxy"
Aug 01 00:16:10.615912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615826    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerName="kserve-container"
Aug 01 00:16:10.615912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.615831    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="649b43d3-2f38-4a73-b5fa-3f7f89b18641" containerName="kube-rbac-proxy"
Aug 01 00:16:10.618809 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.618792    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.621445 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.621425    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Aug 01 00:16:10.621522 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.621450    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Aug 01 00:16:10.629942 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.629919    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"]
Aug 01 00:16:10.705806 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.705779    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:10.753815 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.753781    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e711c08-3106-4f85-8169-38a1f5be2add-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.753948 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.753835    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e711c08-3106-4f85-8169-38a1f5be2add-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.753948 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.753855    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njg78\" (UniqueName: \"kubernetes.io/projected/2e711c08-3106-4f85-8169-38a1f5be2add-kube-api-access-njg78\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.753948 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.753895    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e711c08-3106-4f85-8169-38a1f5be2add-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.753948 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.753927    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e711c08-3106-4f85-8169-38a1f5be2add-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.855389 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.855306    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e711c08-3106-4f85-8169-38a1f5be2add-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.855389 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.855378    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e711c08-3106-4f85-8169-38a1f5be2add-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.855596 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.855417    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e711c08-3106-4f85-8169-38a1f5be2add-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.855596 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.855440    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njg78\" (UniqueName: \"kubernetes.io/projected/2e711c08-3106-4f85-8169-38a1f5be2add-kube-api-access-njg78\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.855596 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.855461    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e711c08-3106-4f85-8169-38a1f5be2add-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.855817 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.855770    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e711c08-3106-4f85-8169-38a1f5be2add-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.856057 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.856035    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e711c08-3106-4f85-8169-38a1f5be2add-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.856101 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.856073    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e711c08-3106-4f85-8169-38a1f5be2add-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.857799 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.857781    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e711c08-3106-4f85-8169-38a1f5be2add-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.863523 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.863505    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njg78\" (UniqueName: \"kubernetes.io/projected/2e711c08-3106-4f85-8169-38a1f5be2add-kube-api-access-njg78\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:10.928819 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:10.928786    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:11.060356 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:11.060325    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"]
Aug 01 00:16:11.062900 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:16:11.062860    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e711c08_3106_4f85_8169_38a1f5be2add.slice/crio-a18f3a65e2e71dbf6100a4935ad7ea32bae67a2c27aa4a24b31e2cda0a8d7f6c WatchSource:0}: Error finding container a18f3a65e2e71dbf6100a4935ad7ea32bae67a2c27aa4a24b31e2cda0a8d7f6c: Status 404 returned error can't find the container with id a18f3a65e2e71dbf6100a4935ad7ea32bae67a2c27aa4a24b31e2cda0a8d7f6c
Aug 01 00:16:11.947874 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:11.947837    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp" event={"ID":"2e711c08-3106-4f85-8169-38a1f5be2add","Type":"ContainerStarted","Data":"5833dd5e4a1b1f35a92e4bdcc985646920eddd81f4a2a2f96d32fafd2b6e9f3e"}
Aug 01 00:16:11.948270 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:11.947927    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp" event={"ID":"2e711c08-3106-4f85-8169-38a1f5be2add","Type":"ContainerStarted","Data":"a18f3a65e2e71dbf6100a4935ad7ea32bae67a2c27aa4a24b31e2cda0a8d7f6c"}
Aug 01 00:16:14.963011 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:14.962912    2546 generic.go:356] "Generic (PLEG): container finished" podID="2e711c08-3106-4f85-8169-38a1f5be2add" containerID="5833dd5e4a1b1f35a92e4bdcc985646920eddd81f4a2a2f96d32fafd2b6e9f3e" exitCode=0
Aug 01 00:16:14.963011 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:14.962987    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp" event={"ID":"2e711c08-3106-4f85-8169-38a1f5be2add","Type":"ContainerDied","Data":"5833dd5e4a1b1f35a92e4bdcc985646920eddd81f4a2a2f96d32fafd2b6e9f3e"}
Aug 01 00:16:15.970492 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:15.970458    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp" event={"ID":"2e711c08-3106-4f85-8169-38a1f5be2add","Type":"ContainerStarted","Data":"13488e72892f752020a3475be3ba6bb1f24c4f079ad4fcfc8cdcb5a481a24cd4"}
Aug 01 00:16:15.970871 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:15.970521    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp" event={"ID":"2e711c08-3106-4f85-8169-38a1f5be2add","Type":"ContainerStarted","Data":"65ee2783d43c73bd0bb664b9fab7c7129b5b34eaaabbcef60ab9475f57814fe7"}
Aug 01 00:16:15.970871 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:15.970713    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:15.970871 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:15.970764    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:15.991050 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:15.991004    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp" podStartSLOduration=2.443148351 podStartE2EDuration="5.990989483s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.547841132s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:16:15.990010236 +0000 UTC m=+1806.449473059" watchObservedRunningTime="2026-08-01 00:16:15.990989483 +0000 UTC m=+1806.450452295"
Aug 01 00:16:19.581432 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:19.581396    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:16:21.267946 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:21.267912    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"]
Aug 01 00:16:21.268526 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:21.268177    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kserve-container" containerID="cri-o://277f1d5f0c1d4e6a0939aa476f4801d816d96ce82e7d1386701b34c14a715349" gracePeriod=30
Aug 01 00:16:21.268526 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:21.268201    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kube-rbac-proxy" containerID="cri-o://6a70439b0a76f457fad3f2ef32eee3b6f01ccfed0f52b51ac337705e14fd8a78" gracePeriod=30
Aug 01 00:16:21.272775 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:21.272749    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"]
Aug 01 00:16:21.273047 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:21.273003    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kserve-container" containerID="cri-o://be58b35041e8265f21b82d7bc865ab59bd93f5b37a4ffdf76fed16356b5008af" gracePeriod=30
Aug 01 00:16:21.273142 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:21.273057    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kube-rbac-proxy" containerID="cri-o://a842cbd83969ecd02b2ce2487c6bd55789648121001fc825b99e7720024e5c02" gracePeriod=30
Aug 01 00:16:21.288241 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:21.288184    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:21.344951 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:21.344917    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:21.981091 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:21.981062    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:16:21.998502 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:21.998473    2546 generic.go:356] "Generic (PLEG): container finished" podID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerID="6a70439b0a76f457fad3f2ef32eee3b6f01ccfed0f52b51ac337705e14fd8a78" exitCode=2
Aug 01 00:16:21.998652 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:21.998553    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" event={"ID":"cd7cbdd3-f124-47d7-8e03-b9448bf14770","Type":"ContainerDied","Data":"6a70439b0a76f457fad3f2ef32eee3b6f01ccfed0f52b51ac337705e14fd8a78"}
Aug 01 00:16:22.001001 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:22.000974    2546 generic.go:356] "Generic (PLEG): container finished" podID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerID="a842cbd83969ecd02b2ce2487c6bd55789648121001fc825b99e7720024e5c02" exitCode=2
Aug 01 00:16:22.001122 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:22.001046    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" event={"ID":"a56aabd0-e071-4551-b21b-72d6688b23dd","Type":"ContainerDied","Data":"a842cbd83969ecd02b2ce2487c6bd55789648121001fc825b99e7720024e5c02"}
Aug 01 00:16:24.576695 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:24.576653    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Aug 01 00:16:24.577102 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:24.576661    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Aug 01 00:16:26.022657 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.022613    2546 generic.go:356] "Generic (PLEG): container finished" podID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerID="277f1d5f0c1d4e6a0939aa476f4801d816d96ce82e7d1386701b34c14a715349" exitCode=0
Aug 01 00:16:26.023097 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.022719    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" event={"ID":"cd7cbdd3-f124-47d7-8e03-b9448bf14770","Type":"ContainerDied","Data":"277f1d5f0c1d4e6a0939aa476f4801d816d96ce82e7d1386701b34c14a715349"}
Aug 01 00:16:26.025564 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.025450    2546 generic.go:356] "Generic (PLEG): container finished" podID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerID="be58b35041e8265f21b82d7bc865ab59bd93f5b37a4ffdf76fed16356b5008af" exitCode=0
Aug 01 00:16:26.025564 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.025525    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" event={"ID":"a56aabd0-e071-4551-b21b-72d6688b23dd","Type":"ContainerDied","Data":"be58b35041e8265f21b82d7bc865ab59bd93f5b37a4ffdf76fed16356b5008af"}
Aug 01 00:16:26.144319 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.144292    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:16:26.147107 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.147086    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:16:26.294414 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.294335    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a56aabd0-e071-4551-b21b-72d6688b23dd-kube-rbac-proxy-sar-config\") pod \"a56aabd0-e071-4551-b21b-72d6688b23dd\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") "
Aug 01 00:16:26.294414 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.294374    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a56aabd0-e071-4551-b21b-72d6688b23dd-proxy-tls\") pod \"a56aabd0-e071-4551-b21b-72d6688b23dd\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") "
Aug 01 00:16:26.294414 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.294397    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a56aabd0-e071-4551-b21b-72d6688b23dd-cabundle-cert\") pod \"a56aabd0-e071-4551-b21b-72d6688b23dd\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") "
Aug 01 00:16:26.294701 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.294426    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kube-rbac-proxy-sar-config\") pod \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") "
Aug 01 00:16:26.294701 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.294458    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd7cbdd3-f124-47d7-8e03-b9448bf14770-cabundle-cert\") pod \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") "
Aug 01 00:16:26.294701 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.294485    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a56aabd0-e071-4551-b21b-72d6688b23dd-kserve-provision-location\") pod \"a56aabd0-e071-4551-b21b-72d6688b23dd\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") "
Aug 01 00:16:26.294701 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.294548    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh7gl\" (UniqueName: \"kubernetes.io/projected/a56aabd0-e071-4551-b21b-72d6688b23dd-kube-api-access-gh7gl\") pod \"a56aabd0-e071-4551-b21b-72d6688b23dd\" (UID: \"a56aabd0-e071-4551-b21b-72d6688b23dd\") "
Aug 01 00:16:26.294701 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.294576    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kserve-provision-location\") pod \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") "
Aug 01 00:16:26.294701 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.294598    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd7cbdd3-f124-47d7-8e03-b9448bf14770-proxy-tls\") pod \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") "
Aug 01 00:16:26.294701 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.294622    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtvzg\" (UniqueName: \"kubernetes.io/projected/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kube-api-access-rtvzg\") pod \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\" (UID: \"cd7cbdd3-f124-47d7-8e03-b9448bf14770\") "
Aug 01 00:16:26.295051 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.294801    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a56aabd0-e071-4551-b21b-72d6688b23dd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a56aabd0-e071-4551-b21b-72d6688b23dd" (UID: "a56aabd0-e071-4551-b21b-72d6688b23dd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:16:26.295051 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.294831    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a56aabd0-e071-4551-b21b-72d6688b23dd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a56aabd0-e071-4551-b21b-72d6688b23dd" (UID: "a56aabd0-e071-4551-b21b-72d6688b23dd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:16:26.295051 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.294864    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56aabd0-e071-4551-b21b-72d6688b23dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a56aabd0-e071-4551-b21b-72d6688b23dd" (UID: "a56aabd0-e071-4551-b21b-72d6688b23dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:16:26.295051 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.294803    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cd7cbdd3-f124-47d7-8e03-b9448bf14770" (UID: "cd7cbdd3-f124-47d7-8e03-b9448bf14770"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:16:26.295051 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.294871    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7cbdd3-f124-47d7-8e03-b9448bf14770-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd7cbdd3-f124-47d7-8e03-b9448bf14770" (UID: "cd7cbdd3-f124-47d7-8e03-b9448bf14770"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:16:26.295051 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.295038    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd7cbdd3-f124-47d7-8e03-b9448bf14770" (UID: "cd7cbdd3-f124-47d7-8e03-b9448bf14770"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:16:26.295364 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.295059    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a56aabd0-e071-4551-b21b-72d6688b23dd-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:16:26.295364 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.295075    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:16:26.295364 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.295085    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd7cbdd3-f124-47d7-8e03-b9448bf14770-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:16:26.295364 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.295094    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a56aabd0-e071-4551-b21b-72d6688b23dd-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:16:26.295364 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.295107    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a56aabd0-e071-4551-b21b-72d6688b23dd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:16:26.297014 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.296976    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a56aabd0-e071-4551-b21b-72d6688b23dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a56aabd0-e071-4551-b21b-72d6688b23dd" (UID: "a56aabd0-e071-4551-b21b-72d6688b23dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:16:26.297130 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.297003    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kube-api-access-rtvzg" (OuterVolumeSpecName: "kube-api-access-rtvzg") pod "cd7cbdd3-f124-47d7-8e03-b9448bf14770" (UID: "cd7cbdd3-f124-47d7-8e03-b9448bf14770"). InnerVolumeSpecName "kube-api-access-rtvzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:16:26.297130 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.297019    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7cbdd3-f124-47d7-8e03-b9448bf14770-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd7cbdd3-f124-47d7-8e03-b9448bf14770" (UID: "cd7cbdd3-f124-47d7-8e03-b9448bf14770"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:16:26.297130 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.297037    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56aabd0-e071-4551-b21b-72d6688b23dd-kube-api-access-gh7gl" (OuterVolumeSpecName: "kube-api-access-gh7gl") pod "a56aabd0-e071-4551-b21b-72d6688b23dd" (UID: "a56aabd0-e071-4551-b21b-72d6688b23dd"). InnerVolumeSpecName "kube-api-access-gh7gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:16:26.396132 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.396104    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a56aabd0-e071-4551-b21b-72d6688b23dd-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:16:26.396132 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.396130    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh7gl\" (UniqueName: \"kubernetes.io/projected/a56aabd0-e071-4551-b21b-72d6688b23dd-kube-api-access-gh7gl\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:16:26.396321 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.396141    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:16:26.396321 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.396149    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd7cbdd3-f124-47d7-8e03-b9448bf14770-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:16:26.396321 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.396159    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtvzg\" (UniqueName: \"kubernetes.io/projected/cd7cbdd3-f124-47d7-8e03-b9448bf14770-kube-api-access-rtvzg\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:16:26.398024 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.398001    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:26.431995 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.431965    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"]
Aug 01 00:16:26.432316 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432304    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kube-rbac-proxy"
Aug 01 00:16:26.432365 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432317    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kube-rbac-proxy"
Aug 01 00:16:26.432365 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432331    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="storage-initializer"
Aug 01 00:16:26.432365 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432337    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="storage-initializer"
Aug 01 00:16:26.432365 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432346    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kube-rbac-proxy"
Aug 01 00:16:26.432365 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432354    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kube-rbac-proxy"
Aug 01 00:16:26.432529 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432368    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="storage-initializer"
Aug 01 00:16:26.432529 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432374    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="storage-initializer"
Aug 01 00:16:26.432529 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432381    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kserve-container"
Aug 01 00:16:26.432529 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432386    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kserve-container"
Aug 01 00:16:26.432529 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432393    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kserve-container"
Aug 01 00:16:26.432529 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432398    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kserve-container"
Aug 01 00:16:26.432529 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432449    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kserve-container"
Aug 01 00:16:26.432529 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432456    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" containerName="kube-rbac-proxy"
Aug 01 00:16:26.432529 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432462    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kserve-container"
Aug 01 00:16:26.432529 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.432469    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" containerName="kube-rbac-proxy"
Aug 01 00:16:26.436778 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.436762    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:26.439202 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.439182    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Aug 01 00:16:26.439506 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.439492    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Aug 01 00:16:26.444819 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.444801    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"]
Aug 01 00:16:26.466393 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.466369    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"]
Aug 01 00:16:26.468402 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.468374    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:26.469935 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.469920    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:26.472179 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.472162    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Aug 01 00:16:26.479437 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.479417    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"]
Aug 01 00:16:26.597354 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.597288    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:26.597354 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.597325    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b62a662-f4c1-4293-b374-2a62c06c2d07-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:26.597502 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.597364    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdqpk\" (UniqueName: \"kubernetes.io/projected/6b62a662-f4c1-4293-b374-2a62c06c2d07-kube-api-access-gdqpk\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:26.597502 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.597385    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-cabundle-cert\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:26.597502 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.597405    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfnwz\" (UniqueName: \"kubernetes.io/projected/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kube-api-access-pfnwz\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:26.597502 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.597464    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b62a662-f4c1-4293-b374-2a62c06c2d07-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:26.597634 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.597505    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b62a662-f4c1-4293-b374-2a62c06c2d07-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:26.597634 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.597526    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-proxy-tls\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:26.597634 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.597547    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kserve-provision-location\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:26.597634 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.597564    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b62a662-f4c1-4293-b374-2a62c06c2d07-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:26.698192 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.698166    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b62a662-f4c1-4293-b374-2a62c06c2d07-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:26.698330 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.698200    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b62a662-f4c1-4293-b374-2a62c06c2d07-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:26.698330 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.698219    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-proxy-tls\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:26.698330 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.698240    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kserve-provision-location\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:26.698458 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:16:26.698346    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Aug 01 00:16:26.698458 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.698375    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b62a662-f4c1-4293-b374-2a62c06c2d07-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:26.698458 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:16:26.698399    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-proxy-tls podName:2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2 nodeName:}" failed. No retries permitted until 2026-08-01 00:16:27.198383974 +0000 UTC m=+1817.657846764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-proxy-tls") pod "isvc-canary-promote-predictor-569b87b6dc-wfvzb" (UID: "2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2") : secret "isvc-canary-promote-predictor-serving-cert" not found
Aug 01 00:16:26.698458 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:16:26.698351    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Aug 01 00:16:26.698458 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.698444    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:26.698708 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.698485    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b62a662-f4c1-4293-b374-2a62c06c2d07-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:26.698708 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:16:26.698507    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b62a662-f4c1-4293-b374-2a62c06c2d07-proxy-tls podName:6b62a662-f4c1-4293-b374-2a62c06c2d07 nodeName:}" failed. No retries permitted until 2026-08-01 00:16:27.198484647 +0000 UTC m=+1817.657947459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6b62a662-f4c1-4293-b374-2a62c06c2d07-proxy-tls") pod "isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" (UID: "6b62a662-f4c1-4293-b374-2a62c06c2d07") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Aug 01 00:16:26.698708 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.698610    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b62a662-f4c1-4293-b374-2a62c06c2d07-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:26.698708 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.698618    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdqpk\" (UniqueName: \"kubernetes.io/projected/6b62a662-f4c1-4293-b374-2a62c06c2d07-kube-api-access-gdqpk\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:26.698708 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.698624    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kserve-provision-location\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:26.698708 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.698680    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-cabundle-cert\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:26.699020 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.698716    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfnwz\" (UniqueName: \"kubernetes.io/projected/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kube-api-access-pfnwz\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:26.699081 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.699069    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b62a662-f4c1-4293-b374-2a62c06c2d07-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:26.699283 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.699241    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b62a662-f4c1-4293-b374-2a62c06c2d07-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:26.699359 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.699241    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:26.699359 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.699307    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-cabundle-cert\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:26.706889 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.706870    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfnwz\" (UniqueName: \"kubernetes.io/projected/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kube-api-access-pfnwz\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:26.706968 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.706922    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdqpk\" (UniqueName: \"kubernetes.io/projected/6b62a662-f4c1-4293-b374-2a62c06c2d07-kube-api-access-gdqpk\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:26.839389 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:26.839354    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:27.031408 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.031384    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"
Aug 01 00:16:27.031830 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.031382    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s" event={"ID":"cd7cbdd3-f124-47d7-8e03-b9448bf14770","Type":"ContainerDied","Data":"d609fe5ce040cbf20e838ca76ddec6e6e4af76d37cbd0686907cf4d02a51483a"}
Aug 01 00:16:27.031830 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.031555    2546 scope.go:117] "RemoveContainer" containerID="6a70439b0a76f457fad3f2ef32eee3b6f01ccfed0f52b51ac337705e14fd8a78"
Aug 01 00:16:27.033649 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.033631    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"
Aug 01 00:16:27.033769 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.033632    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv" event={"ID":"a56aabd0-e071-4551-b21b-72d6688b23dd","Type":"ContainerDied","Data":"f264867bc9ecdfaa8066451235408dbd2703336a53a1f9c7f85ca3b1bd3114e4"}
Aug 01 00:16:27.040179 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.040163    2546 scope.go:117] "RemoveContainer" containerID="277f1d5f0c1d4e6a0939aa476f4801d816d96ce82e7d1386701b34c14a715349"
Aug 01 00:16:27.048913 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.048895    2546 scope.go:117] "RemoveContainer" containerID="48b9731584bd42c5bbfe09849d8add9e94714a8019628d4aeb97513d9a57c59d"
Aug 01 00:16:27.058298 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.058276    2546 scope.go:117] "RemoveContainer" containerID="a842cbd83969ecd02b2ce2487c6bd55789648121001fc825b99e7720024e5c02"
Aug 01 00:16:27.058620 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.058602    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"]
Aug 01 00:16:27.063095 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.063072    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-c4c849674-7fk4s"]
Aug 01 00:16:27.066475 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.066443    2546 scope.go:117] "RemoveContainer" containerID="be58b35041e8265f21b82d7bc865ab59bd93f5b37a4ffdf76fed16356b5008af"
Aug 01 00:16:27.072592 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.072572    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"]
Aug 01 00:16:27.073764 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.073751    2546 scope.go:117] "RemoveContainer" containerID="7802e2467f36a627ee671c7147a1703070adbd381f00008126a9d0979891bb0f"
Aug 01 00:16:27.076646 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.076624    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8644bc947f-8rzrv"]
Aug 01 00:16:27.203648 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.203563    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b62a662-f4c1-4293-b374-2a62c06c2d07-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:27.203648 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.203612    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-proxy-tls\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:27.203856 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:16:27.203747    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Aug 01 00:16:27.203856 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:16:27.203826    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-proxy-tls podName:2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2 nodeName:}" failed. No retries permitted until 2026-08-01 00:16:28.203806972 +0000 UTC m=+1818.663269765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-proxy-tls") pod "isvc-canary-promote-predictor-569b87b6dc-wfvzb" (UID: "2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2") : secret "isvc-canary-promote-predictor-serving-cert" not found
Aug 01 00:16:27.205978 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.205945    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b62a662-f4c1-4293-b374-2a62c06c2d07-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7b47596d4d-4vclh\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:27.254190 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.254160    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:16:27.379965 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.379927    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:27.511794 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:27.511742    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"]
Aug 01 00:16:27.514392 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:16:27.514365    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b62a662_f4c1_4293_b374_2a62c06c2d07.slice/crio-b22cde366106176a7054c137683ea1ed661cbb6f355735113ab7542909655125 WatchSource:0}: Error finding container b22cde366106176a7054c137683ea1ed661cbb6f355735113ab7542909655125: Status 404 returned error can't find the container with id b22cde366106176a7054c137683ea1ed661cbb6f355735113ab7542909655125
Aug 01 00:16:28.039056 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:28.039014    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" event={"ID":"6b62a662-f4c1-4293-b374-2a62c06c2d07","Type":"ContainerStarted","Data":"a8be7767f2aee512d2e7132580be9020859a14622022979d4e181f20995c8d92"}
Aug 01 00:16:28.039552 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:28.039090    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" event={"ID":"6b62a662-f4c1-4293-b374-2a62c06c2d07","Type":"ContainerStarted","Data":"b22cde366106176a7054c137683ea1ed661cbb6f355735113ab7542909655125"}
Aug 01 00:16:28.075635 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:28.075601    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a56aabd0-e071-4551-b21b-72d6688b23dd" path="/var/lib/kubelet/pods/a56aabd0-e071-4551-b21b-72d6688b23dd/volumes"
Aug 01 00:16:28.076185 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:28.076163    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd7cbdd3-f124-47d7-8e03-b9448bf14770" path="/var/lib/kubelet/pods/cd7cbdd3-f124-47d7-8e03-b9448bf14770/volumes"
Aug 01 00:16:28.212438 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:28.212390    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-proxy-tls\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:28.214881 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:28.214857    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-proxy-tls\") pod \"isvc-canary-promote-predictor-569b87b6dc-wfvzb\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:28.247826 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:28.247797    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:28.383569 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:28.383539    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"]
Aug 01 00:16:28.385360 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:16:28.385334    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5b1c45_5e2d_4475_8d59_214ab5c1b7b2.slice/crio-1ec9ce5cbefafcc8d5cbf538595ad21cdbd6b18d5387d8ba5fde72ab2387cd96 WatchSource:0}: Error finding container 1ec9ce5cbefafcc8d5cbf538595ad21cdbd6b18d5387d8ba5fde72ab2387cd96: Status 404 returned error can't find the container with id 1ec9ce5cbefafcc8d5cbf538595ad21cdbd6b18d5387d8ba5fde72ab2387cd96
Aug 01 00:16:29.046178 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:29.046139    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" event={"ID":"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2","Type":"ContainerStarted","Data":"e32eccc2542c23aeb92a860a237e7c14b67bd7ef109136fd7cc3e6787669a7e4"}
Aug 01 00:16:29.046674 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:29.046208    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" event={"ID":"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2","Type":"ContainerStarted","Data":"1ec9ce5cbefafcc8d5cbf538595ad21cdbd6b18d5387d8ba5fde72ab2387cd96"}
Aug 01 00:16:31.055912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:31.055881    2546 generic.go:356] "Generic (PLEG): container finished" podID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerID="a8be7767f2aee512d2e7132580be9020859a14622022979d4e181f20995c8d92" exitCode=0
Aug 01 00:16:31.056223 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:31.055916    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" event={"ID":"6b62a662-f4c1-4293-b374-2a62c06c2d07","Type":"ContainerDied","Data":"a8be7767f2aee512d2e7132580be9020859a14622022979d4e181f20995c8d92"}
Aug 01 00:16:32.062422 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:32.062386    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" event={"ID":"6b62a662-f4c1-4293-b374-2a62c06c2d07","Type":"ContainerStarted","Data":"f73ce78f7ae2e9a4573d1bb0775cc013144f3376b78e30d70fcfd978cf1cdb8b"}
Aug 01 00:16:32.062835 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:32.062473    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" event={"ID":"6b62a662-f4c1-4293-b374-2a62c06c2d07","Type":"ContainerStarted","Data":"93cb5938f1b6c338d438584e7384979eeaebb0b3fba8a113dc5ba54e28356250"}
Aug 01 00:16:32.062835 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:32.062760    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:32.062835 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:32.062819    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:32.063790 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:32.063763    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:16:33.067957 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:33.067921    2546 generic.go:356] "Generic (PLEG): container finished" podID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerID="e32eccc2542c23aeb92a860a237e7c14b67bd7ef109136fd7cc3e6787669a7e4" exitCode=0
Aug 01 00:16:33.070319 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:33.067995    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" event={"ID":"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2","Type":"ContainerDied","Data":"e32eccc2542c23aeb92a860a237e7c14b67bd7ef109136fd7cc3e6787669a7e4"}
Aug 01 00:16:33.070319 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:33.068573    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:16:33.088922 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:33.088880    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" podStartSLOduration=3.6688953399999997 podStartE2EDuration="7.08886549s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.41997015s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:16:32.101814323 +0000 UTC m=+1822.561277135" watchObservedRunningTime="2026-08-01 00:16:33.08886549 +0000 UTC m=+1823.548328332"
Aug 01 00:16:34.075797 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:34.075758    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" event={"ID":"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2","Type":"ContainerStarted","Data":"f11e39546977defe39d6694817c53fd058c8f16bb57eb0b6b8d308f5e94b2a52"}
Aug 01 00:16:34.076182 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:34.075816    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" event={"ID":"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2","Type":"ContainerStarted","Data":"f9346d4f801eb799b5a0e8d0cb1004091204f88943aab6b8fdd4600a89b79fbf"}
Aug 01 00:16:34.076182 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:34.076125    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:34.076286 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:34.076270    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:34.077656 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:34.077631    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Aug 01 00:16:34.096217 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:34.096172    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" podStartSLOduration=4.370356952 podStartE2EDuration="8.096159379s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.725802427s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:16:34.094044413 +0000 UTC m=+1824.553507249" watchObservedRunningTime="2026-08-01 00:16:34.096159379 +0000 UTC m=+1824.555622192"
Aug 01 00:16:35.080188 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:35.080149    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Aug 01 00:16:36.084062 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:36.084024    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Aug 01 00:16:38.078672 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:38.078644    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:16:38.079154 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:38.079130    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:16:41.088608 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:41.088578    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:16:41.089129 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:41.089100    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Aug 01 00:16:48.080040 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:48.079990    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:16:51.089943 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:51.089898    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Aug 01 00:16:52.020867 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:52.020794    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp" podUID="2e711c08-3106-4f85-8169-38a1f5be2add" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Aug 01 00:16:58.079512 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:16:58.079471    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:17:01.089428 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:01.089392    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Aug 01 00:17:01.984175 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:01.984147    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:17:08.079457 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:08.079420    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:17:10.696018 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:10.695982    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"]
Aug 01 00:17:10.696501 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:10.696291    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp" podUID="2e711c08-3106-4f85-8169-38a1f5be2add" containerName="kserve-container" containerID="cri-o://65ee2783d43c73bd0bb664b9fab7c7129b5b34eaaabbcef60ab9475f57814fe7" gracePeriod=30
Aug 01 00:17:10.696501 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:10.696312    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp" podUID="2e711c08-3106-4f85-8169-38a1f5be2add" containerName="kube-rbac-proxy" containerID="cri-o://13488e72892f752020a3475be3ba6bb1f24c4f079ad4fcfc8cdcb5a481a24cd4" gracePeriod=30
Aug 01 00:17:10.721029 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:10.720992    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:11.089082 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:11.088991    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Aug 01 00:17:11.243885 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:11.243854    2546 generic.go:356] "Generic (PLEG): container finished" podID="2e711c08-3106-4f85-8169-38a1f5be2add" containerID="13488e72892f752020a3475be3ba6bb1f24c4f079ad4fcfc8cdcb5a481a24cd4" exitCode=2
Aug 01 00:17:11.244071 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:11.243933    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp" event={"ID":"2e711c08-3106-4f85-8169-38a1f5be2add","Type":"ContainerDied","Data":"13488e72892f752020a3475be3ba6bb1f24c4f079ad4fcfc8cdcb5a481a24cd4"}
Aug 01 00:17:11.975630 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:11.975592    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp" podUID="2e711c08-3106-4f85-8169-38a1f5be2add" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Aug 01 00:17:13.023472 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:13.023422    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp" podUID="2e711c08-3106-4f85-8169-38a1f5be2add" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.46:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Aug 01 00:17:15.840833 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:15.840789    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:15.870446 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:15.870414    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"]
Aug 01 00:17:15.873944 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:15.873929    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:15.876145 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:15.876128    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Aug 01 00:17:15.876284 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:15.876149    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Aug 01 00:17:15.882442 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:15.882419    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"]
Aug 01 00:17:15.936234 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:15.936191    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbc10131-3c23-4348-82b2-b23c57e2de8b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:15.936413 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:15.936288    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc10131-3c23-4348-82b2-b23c57e2de8b-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:15.936413 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:15.936311    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbc10131-3c23-4348-82b2-b23c57e2de8b-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:15.936413 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:15.936388    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbc10131-3c23-4348-82b2-b23c57e2de8b-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:15.936413 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:15.936411    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xmsr\" (UniqueName: \"kubernetes.io/projected/cbc10131-3c23-4348-82b2-b23c57e2de8b-kube-api-access-2xmsr\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:16.037023 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:16.036986    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbc10131-3c23-4348-82b2-b23c57e2de8b-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:16.037023 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:16.037030    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xmsr\" (UniqueName: \"kubernetes.io/projected/cbc10131-3c23-4348-82b2-b23c57e2de8b-kube-api-access-2xmsr\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:16.037309 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:16.037069    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbc10131-3c23-4348-82b2-b23c57e2de8b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:16.037309 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:16.037105    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc10131-3c23-4348-82b2-b23c57e2de8b-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:16.037309 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:16.037124    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbc10131-3c23-4348-82b2-b23c57e2de8b-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:16.037309 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:17:16.037249    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Aug 01 00:17:16.037535 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:17:16.037336    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc10131-3c23-4348-82b2-b23c57e2de8b-proxy-tls podName:cbc10131-3c23-4348-82b2-b23c57e2de8b nodeName:}" failed. No retries permitted until 2026-08-01 00:17:16.5373153 +0000 UTC m=+1866.996778097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cbc10131-3c23-4348-82b2-b23c57e2de8b-proxy-tls") pod "isvc-sklearn-v2-predictor-7bffcff858-p4vzp" (UID: "cbc10131-3c23-4348-82b2-b23c57e2de8b") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Aug 01 00:17:16.037535 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:16.037523    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbc10131-3c23-4348-82b2-b23c57e2de8b-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:16.037750 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:16.037732    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbc10131-3c23-4348-82b2-b23c57e2de8b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:16.037810 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:16.037793    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbc10131-3c23-4348-82b2-b23c57e2de8b-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:16.047568 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:16.047551    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xmsr\" (UniqueName: \"kubernetes.io/projected/cbc10131-3c23-4348-82b2-b23c57e2de8b-kube-api-access-2xmsr\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:16.211426 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:16.211391    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:16.542461 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:16.542352    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc10131-3c23-4348-82b2-b23c57e2de8b-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:16.544809 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:16.544791    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc10131-3c23-4348-82b2-b23c57e2de8b-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7bffcff858-p4vzp\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:16.785371 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:16.785329    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:16.916938 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:16.916915    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"]
Aug 01 00:17:16.919177 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:17:16.919146    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc10131_3c23_4348_82b2_b23c57e2de8b.slice/crio-d5236a5b47a8dbd2e6505b259308cb09eaabb1e55f48534ba5faa9c95218feff WatchSource:0}: Error finding container d5236a5b47a8dbd2e6505b259308cb09eaabb1e55f48534ba5faa9c95218feff: Status 404 returned error can't find the container with id d5236a5b47a8dbd2e6505b259308cb09eaabb1e55f48534ba5faa9c95218feff
Aug 01 00:17:16.975953 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:16.975918    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp" podUID="2e711c08-3106-4f85-8169-38a1f5be2add" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Aug 01 00:17:17.271592 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:17.271557    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" event={"ID":"cbc10131-3c23-4348-82b2-b23c57e2de8b","Type":"ContainerStarted","Data":"fc7ab9691599989496958d2175dcc536029f011853cab016c920edeb00911333"}
Aug 01 00:17:17.271789 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:17.271652    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" event={"ID":"cbc10131-3c23-4348-82b2-b23c57e2de8b","Type":"ContainerStarted","Data":"d5236a5b47a8dbd2e6505b259308cb09eaabb1e55f48534ba5faa9c95218feff"}
Aug 01 00:17:18.079694 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.079657    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:17:18.755299 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.755275    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:17:18.762002 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.761982    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e711c08-3106-4f85-8169-38a1f5be2add-proxy-tls\") pod \"2e711c08-3106-4f85-8169-38a1f5be2add\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") "
Aug 01 00:17:18.762108 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.762011    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e711c08-3106-4f85-8169-38a1f5be2add-cabundle-cert\") pod \"2e711c08-3106-4f85-8169-38a1f5be2add\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") "
Aug 01 00:17:18.762108 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.762036    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e711c08-3106-4f85-8169-38a1f5be2add-kserve-provision-location\") pod \"2e711c08-3106-4f85-8169-38a1f5be2add\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") "
Aug 01 00:17:18.762108 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.762062    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njg78\" (UniqueName: \"kubernetes.io/projected/2e711c08-3106-4f85-8169-38a1f5be2add-kube-api-access-njg78\") pod \"2e711c08-3106-4f85-8169-38a1f5be2add\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") "
Aug 01 00:17:18.762108 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.762101    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e711c08-3106-4f85-8169-38a1f5be2add-kube-rbac-proxy-sar-config\") pod \"2e711c08-3106-4f85-8169-38a1f5be2add\" (UID: \"2e711c08-3106-4f85-8169-38a1f5be2add\") "
Aug 01 00:17:18.762368 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.762350    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e711c08-3106-4f85-8169-38a1f5be2add-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e711c08-3106-4f85-8169-38a1f5be2add" (UID: "2e711c08-3106-4f85-8169-38a1f5be2add"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:17:18.762430 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.762350    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e711c08-3106-4f85-8169-38a1f5be2add-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e711c08-3106-4f85-8169-38a1f5be2add" (UID: "2e711c08-3106-4f85-8169-38a1f5be2add"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:17:18.762485 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.762437    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e711c08-3106-4f85-8169-38a1f5be2add-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:17:18.762485 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.762461    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e711c08-3106-4f85-8169-38a1f5be2add-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:17:18.762600 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.762525    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e711c08-3106-4f85-8169-38a1f5be2add-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e711c08-3106-4f85-8169-38a1f5be2add" (UID: "2e711c08-3106-4f85-8169-38a1f5be2add"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:17:18.764002 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.763981    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e711c08-3106-4f85-8169-38a1f5be2add-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e711c08-3106-4f85-8169-38a1f5be2add" (UID: "2e711c08-3106-4f85-8169-38a1f5be2add"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:17:18.764161 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.764144    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e711c08-3106-4f85-8169-38a1f5be2add-kube-api-access-njg78" (OuterVolumeSpecName: "kube-api-access-njg78") pod "2e711c08-3106-4f85-8169-38a1f5be2add" (UID: "2e711c08-3106-4f85-8169-38a1f5be2add"). InnerVolumeSpecName "kube-api-access-njg78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:17:18.863568 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.863537    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e711c08-3106-4f85-8169-38a1f5be2add-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:17:18.863568 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.863567    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e711c08-3106-4f85-8169-38a1f5be2add-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:17:18.863753 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:18.863578    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njg78\" (UniqueName: \"kubernetes.io/projected/2e711c08-3106-4f85-8169-38a1f5be2add-kube-api-access-njg78\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:17:19.283135 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:19.283044    2546 generic.go:356] "Generic (PLEG): container finished" podID="2e711c08-3106-4f85-8169-38a1f5be2add" containerID="65ee2783d43c73bd0bb664b9fab7c7129b5b34eaaabbcef60ab9475f57814fe7" exitCode=0
Aug 01 00:17:19.283135 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:19.283093    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"
Aug 01 00:17:19.283135 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:19.283119    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp" event={"ID":"2e711c08-3106-4f85-8169-38a1f5be2add","Type":"ContainerDied","Data":"65ee2783d43c73bd0bb664b9fab7c7129b5b34eaaabbcef60ab9475f57814fe7"}
Aug 01 00:17:19.283617 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:19.283186    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp" event={"ID":"2e711c08-3106-4f85-8169-38a1f5be2add","Type":"ContainerDied","Data":"a18f3a65e2e71dbf6100a4935ad7ea32bae67a2c27aa4a24b31e2cda0a8d7f6c"}
Aug 01 00:17:19.283617 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:19.283214    2546 scope.go:117] "RemoveContainer" containerID="13488e72892f752020a3475be3ba6bb1f24c4f079ad4fcfc8cdcb5a481a24cd4"
Aug 01 00:17:19.292325 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:19.292308    2546 scope.go:117] "RemoveContainer" containerID="65ee2783d43c73bd0bb664b9fab7c7129b5b34eaaabbcef60ab9475f57814fe7"
Aug 01 00:17:19.299788 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:19.299773    2546 scope.go:117] "RemoveContainer" containerID="5833dd5e4a1b1f35a92e4bdcc985646920eddd81f4a2a2f96d32fafd2b6e9f3e"
Aug 01 00:17:19.308155 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:19.307670    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"]
Aug 01 00:17:19.310088 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:19.310072    2546 scope.go:117] "RemoveContainer" containerID="13488e72892f752020a3475be3ba6bb1f24c4f079ad4fcfc8cdcb5a481a24cd4"
Aug 01 00:17:19.310363 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:17:19.310334    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13488e72892f752020a3475be3ba6bb1f24c4f079ad4fcfc8cdcb5a481a24cd4\": container with ID starting with 13488e72892f752020a3475be3ba6bb1f24c4f079ad4fcfc8cdcb5a481a24cd4 not found: ID does not exist" containerID="13488e72892f752020a3475be3ba6bb1f24c4f079ad4fcfc8cdcb5a481a24cd4"
Aug 01 00:17:19.310463 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:19.310374    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13488e72892f752020a3475be3ba6bb1f24c4f079ad4fcfc8cdcb5a481a24cd4"} err="failed to get container status \"13488e72892f752020a3475be3ba6bb1f24c4f079ad4fcfc8cdcb5a481a24cd4\": rpc error: code = NotFound desc = could not find container \"13488e72892f752020a3475be3ba6bb1f24c4f079ad4fcfc8cdcb5a481a24cd4\": container with ID starting with 13488e72892f752020a3475be3ba6bb1f24c4f079ad4fcfc8cdcb5a481a24cd4 not found: ID does not exist"
Aug 01 00:17:19.310463 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:19.310395    2546 scope.go:117] "RemoveContainer" containerID="65ee2783d43c73bd0bb664b9fab7c7129b5b34eaaabbcef60ab9475f57814fe7"
Aug 01 00:17:19.310699 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:17:19.310678    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ee2783d43c73bd0bb664b9fab7c7129b5b34eaaabbcef60ab9475f57814fe7\": container with ID starting with 65ee2783d43c73bd0bb664b9fab7c7129b5b34eaaabbcef60ab9475f57814fe7 not found: ID does not exist" containerID="65ee2783d43c73bd0bb664b9fab7c7129b5b34eaaabbcef60ab9475f57814fe7"
Aug 01 00:17:19.310767 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:19.310705    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ee2783d43c73bd0bb664b9fab7c7129b5b34eaaabbcef60ab9475f57814fe7"} err="failed to get container status \"65ee2783d43c73bd0bb664b9fab7c7129b5b34eaaabbcef60ab9475f57814fe7\": rpc error: code = NotFound desc = could not find container \"65ee2783d43c73bd0bb664b9fab7c7129b5b34eaaabbcef60ab9475f57814fe7\": container with ID starting with 65ee2783d43c73bd0bb664b9fab7c7129b5b34eaaabbcef60ab9475f57814fe7 not found: ID does not exist"
Aug 01 00:17:19.310767 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:19.310720    2546 scope.go:117] "RemoveContainer" containerID="5833dd5e4a1b1f35a92e4bdcc985646920eddd81f4a2a2f96d32fafd2b6e9f3e"
Aug 01 00:17:19.311008 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:17:19.310984    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5833dd5e4a1b1f35a92e4bdcc985646920eddd81f4a2a2f96d32fafd2b6e9f3e\": container with ID starting with 5833dd5e4a1b1f35a92e4bdcc985646920eddd81f4a2a2f96d32fafd2b6e9f3e not found: ID does not exist" containerID="5833dd5e4a1b1f35a92e4bdcc985646920eddd81f4a2a2f96d32fafd2b6e9f3e"
Aug 01 00:17:19.311101 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:19.311017    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5833dd5e4a1b1f35a92e4bdcc985646920eddd81f4a2a2f96d32fafd2b6e9f3e"} err="failed to get container status \"5833dd5e4a1b1f35a92e4bdcc985646920eddd81f4a2a2f96d32fafd2b6e9f3e\": rpc error: code = NotFound desc = could not find container \"5833dd5e4a1b1f35a92e4bdcc985646920eddd81f4a2a2f96d32fafd2b6e9f3e\": container with ID starting with 5833dd5e4a1b1f35a92e4bdcc985646920eddd81f4a2a2f96d32fafd2b6e9f3e not found: ID does not exist"
Aug 01 00:17:19.312898 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:19.312880    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-v74mp"]
Aug 01 00:17:20.076051 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:20.076017    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e711c08-3106-4f85-8169-38a1f5be2add" path="/var/lib/kubelet/pods/2e711c08-3106-4f85-8169-38a1f5be2add/volumes"
Aug 01 00:17:21.089239 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:21.089205    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Aug 01 00:17:21.294045 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:21.294012    2546 generic.go:356] "Generic (PLEG): container finished" podID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerID="fc7ab9691599989496958d2175dcc536029f011853cab016c920edeb00911333" exitCode=0
Aug 01 00:17:21.294181 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:21.294068    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" event={"ID":"cbc10131-3c23-4348-82b2-b23c57e2de8b","Type":"ContainerDied","Data":"fc7ab9691599989496958d2175dcc536029f011853cab016c920edeb00911333"}
Aug 01 00:17:22.300102 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:22.300068    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" event={"ID":"cbc10131-3c23-4348-82b2-b23c57e2de8b","Type":"ContainerStarted","Data":"a53f15edcbeec46a79beb1a920b932c56550a82d60ecc90ba5c5e3c1282161b1"}
Aug 01 00:17:22.300516 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:22.300130    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" event={"ID":"cbc10131-3c23-4348-82b2-b23c57e2de8b","Type":"ContainerStarted","Data":"44230cc3b47a94246453beb8cc71a2f7d69f003d4567365724bfa442fd5cc95c"}
Aug 01 00:17:22.300516 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:22.300407    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:22.300626 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:22.300544    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:22.301842 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:22.301814    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Aug 01 00:17:22.320027 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:22.319982    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" podStartSLOduration=3.774748892 podStartE2EDuration="7.319963943s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.545215051s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:17:22.318358291 +0000 UTC m=+1872.777821104" watchObservedRunningTime="2026-08-01 00:17:22.319963943 +0000 UTC m=+1872.779426756"
Aug 01 00:17:23.304929 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:23.304889    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Aug 01 00:17:28.079068 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:28.079030    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:17:28.310073 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:28.310042    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:17:28.310611 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:28.310579    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Aug 01 00:17:31.089031 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:31.088991    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Aug 01 00:17:38.079507 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:38.079470    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:17:38.310695 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:38.310656    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Aug 01 00:17:41.089179 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:41.089131    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Aug 01 00:17:48.080390 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:48.080359    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:17:48.310600 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:48.310562    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Aug 01 00:17:51.089393 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:51.089368    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:17:56.510918 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:56.510887    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"]
Aug 01 00:17:56.511416 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:56.511107    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kserve-container" containerID="cri-o://f9346d4f801eb799b5a0e8d0cb1004091204f88943aab6b8fdd4600a89b79fbf" gracePeriod=30
Aug 01 00:17:56.511416 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:56.511150    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kube-rbac-proxy" containerID="cri-o://f11e39546977defe39d6694817c53fd058c8f16bb57eb0b6b8d308f5e94b2a52" gracePeriod=30
Aug 01 00:17:56.545248 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:56.545225    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:17:57.454861 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:57.454830    2546 generic.go:356] "Generic (PLEG): container finished" podID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerID="f11e39546977defe39d6694817c53fd058c8f16bb57eb0b6b8d308f5e94b2a52" exitCode=2
Aug 01 00:17:57.455028 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:57.454902    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" event={"ID":"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2","Type":"ContainerDied","Data":"f11e39546977defe39d6694817c53fd058c8f16bb57eb0b6b8d308f5e94b2a52"}
Aug 01 00:17:58.310535 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:17:58.310499    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Aug 01 00:18:00.450303 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.450279    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:18:00.469000 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.468941    2546 generic.go:356] "Generic (PLEG): container finished" podID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerID="f9346d4f801eb799b5a0e8d0cb1004091204f88943aab6b8fdd4600a89b79fbf" exitCode=0
Aug 01 00:18:00.469000 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.468992    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"
Aug 01 00:18:00.469182 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.469043    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" event={"ID":"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2","Type":"ContainerDied","Data":"f9346d4f801eb799b5a0e8d0cb1004091204f88943aab6b8fdd4600a89b79fbf"}
Aug 01 00:18:00.469182 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.469128    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb" event={"ID":"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2","Type":"ContainerDied","Data":"1ec9ce5cbefafcc8d5cbf538595ad21cdbd6b18d5387d8ba5fde72ab2387cd96"}
Aug 01 00:18:00.469182 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.469172    2546 scope.go:117] "RemoveContainer" containerID="f11e39546977defe39d6694817c53fd058c8f16bb57eb0b6b8d308f5e94b2a52"
Aug 01 00:18:00.479545 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.479533    2546 scope.go:117] "RemoveContainer" containerID="f9346d4f801eb799b5a0e8d0cb1004091204f88943aab6b8fdd4600a89b79fbf"
Aug 01 00:18:00.488077 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.488058    2546 scope.go:117] "RemoveContainer" containerID="e32eccc2542c23aeb92a860a237e7c14b67bd7ef109136fd7cc3e6787669a7e4"
Aug 01 00:18:00.498224 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.498211    2546 scope.go:117] "RemoveContainer" containerID="f11e39546977defe39d6694817c53fd058c8f16bb57eb0b6b8d308f5e94b2a52"
Aug 01 00:18:00.498543 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:18:00.498521    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f11e39546977defe39d6694817c53fd058c8f16bb57eb0b6b8d308f5e94b2a52\": container with ID starting with f11e39546977defe39d6694817c53fd058c8f16bb57eb0b6b8d308f5e94b2a52 not found: ID does not exist" containerID="f11e39546977defe39d6694817c53fd058c8f16bb57eb0b6b8d308f5e94b2a52"
Aug 01 00:18:00.498639 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.498551    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f11e39546977defe39d6694817c53fd058c8f16bb57eb0b6b8d308f5e94b2a52"} err="failed to get container status \"f11e39546977defe39d6694817c53fd058c8f16bb57eb0b6b8d308f5e94b2a52\": rpc error: code = NotFound desc = could not find container \"f11e39546977defe39d6694817c53fd058c8f16bb57eb0b6b8d308f5e94b2a52\": container with ID starting with f11e39546977defe39d6694817c53fd058c8f16bb57eb0b6b8d308f5e94b2a52 not found: ID does not exist"
Aug 01 00:18:00.498639 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.498573    2546 scope.go:117] "RemoveContainer" containerID="f9346d4f801eb799b5a0e8d0cb1004091204f88943aab6b8fdd4600a89b79fbf"
Aug 01 00:18:00.498941 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:18:00.498897    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9346d4f801eb799b5a0e8d0cb1004091204f88943aab6b8fdd4600a89b79fbf\": container with ID starting with f9346d4f801eb799b5a0e8d0cb1004091204f88943aab6b8fdd4600a89b79fbf not found: ID does not exist" containerID="f9346d4f801eb799b5a0e8d0cb1004091204f88943aab6b8fdd4600a89b79fbf"
Aug 01 00:18:00.498941 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.498931    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9346d4f801eb799b5a0e8d0cb1004091204f88943aab6b8fdd4600a89b79fbf"} err="failed to get container status \"f9346d4f801eb799b5a0e8d0cb1004091204f88943aab6b8fdd4600a89b79fbf\": rpc error: code = NotFound desc = could not find container \"f9346d4f801eb799b5a0e8d0cb1004091204f88943aab6b8fdd4600a89b79fbf\": container with ID starting with f9346d4f801eb799b5a0e8d0cb1004091204f88943aab6b8fdd4600a89b79fbf not found: ID does not exist"
Aug 01 00:18:00.499087 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.498951    2546 scope.go:117] "RemoveContainer" containerID="e32eccc2542c23aeb92a860a237e7c14b67bd7ef109136fd7cc3e6787669a7e4"
Aug 01 00:18:00.499193 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:18:00.499178    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e32eccc2542c23aeb92a860a237e7c14b67bd7ef109136fd7cc3e6787669a7e4\": container with ID starting with e32eccc2542c23aeb92a860a237e7c14b67bd7ef109136fd7cc3e6787669a7e4 not found: ID does not exist" containerID="e32eccc2542c23aeb92a860a237e7c14b67bd7ef109136fd7cc3e6787669a7e4"
Aug 01 00:18:00.499273 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.499196    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e32eccc2542c23aeb92a860a237e7c14b67bd7ef109136fd7cc3e6787669a7e4"} err="failed to get container status \"e32eccc2542c23aeb92a860a237e7c14b67bd7ef109136fd7cc3e6787669a7e4\": rpc error: code = NotFound desc = could not find container \"e32eccc2542c23aeb92a860a237e7c14b67bd7ef109136fd7cc3e6787669a7e4\": container with ID starting with e32eccc2542c23aeb92a860a237e7c14b67bd7ef109136fd7cc3e6787669a7e4 not found: ID does not exist"
Aug 01 00:18:00.616963 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.616942    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kube-rbac-proxy-sar-config\") pod \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") "
Aug 01 00:18:00.617066 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.616983    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kserve-provision-location\") pod \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") "
Aug 01 00:18:00.617066 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.617010    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-cabundle-cert\") pod \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") "
Aug 01 00:18:00.617141 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.617063    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfnwz\" (UniqueName: \"kubernetes.io/projected/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kube-api-access-pfnwz\") pod \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") "
Aug 01 00:18:00.617229 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.617209    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-proxy-tls\") pod \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\" (UID: \"2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2\") "
Aug 01 00:18:00.617335 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.617311    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" (UID: "2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:18:00.617377 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.617318    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" (UID: "2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:18:00.617419 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.617378    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" (UID: "2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:18:00.617553 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.617536    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:18:00.617553 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.617554    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:18:00.617668 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.617563    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:18:00.618933 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.618905    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kube-api-access-pfnwz" (OuterVolumeSpecName: "kube-api-access-pfnwz") pod "2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" (UID: "2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2"). InnerVolumeSpecName "kube-api-access-pfnwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:18:00.619006 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.618951    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" (UID: "2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:18:00.718125 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.718105    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfnwz\" (UniqueName: \"kubernetes.io/projected/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-kube-api-access-pfnwz\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:18:00.718125 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.718126    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:18:00.793086 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.793063    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"]
Aug 01 00:18:00.795675 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:00.795656    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-569b87b6dc-wfvzb"]
Aug 01 00:18:02.075689 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:02.075647    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" path="/var/lib/kubelet/pods/2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2/volumes"
Aug 01 00:18:08.310979 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:08.310941    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Aug 01 00:18:18.311243 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:18.311205    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Aug 01 00:18:28.310533 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:28.310493    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Aug 01 00:18:37.071425 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:37.071389    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:18:46.066355 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:46.066319    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"]
Aug 01 00:18:46.066924 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:46.066542    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kserve-container" containerID="cri-o://44230cc3b47a94246453beb8cc71a2f7d69f003d4567365724bfa442fd5cc95c" gracePeriod=30
Aug 01 00:18:46.066924 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:46.066592    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kube-rbac-proxy" containerID="cri-o://a53f15edcbeec46a79beb1a920b932c56550a82d60ecc90ba5c5e3c1282161b1" gracePeriod=30
Aug 01 00:18:46.085741 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:46.085703    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:46.666653 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:46.666618    2546 generic.go:356] "Generic (PLEG): container finished" podID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerID="a53f15edcbeec46a79beb1a920b932c56550a82d60ecc90ba5c5e3c1282161b1" exitCode=2
Aug 01 00:18:46.666825 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:46.666689    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" event={"ID":"cbc10131-3c23-4348-82b2-b23c57e2de8b","Type":"ContainerDied","Data":"a53f15edcbeec46a79beb1a920b932c56550a82d60ecc90ba5c5e3c1282161b1"}
Aug 01 00:18:47.071108 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:47.071011    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Aug 01 00:18:48.305994 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:48.305938    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Aug 01 00:18:50.686877 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:50.686844    2546 generic.go:356] "Generic (PLEG): container finished" podID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerID="44230cc3b47a94246453beb8cc71a2f7d69f003d4567365724bfa442fd5cc95c" exitCode=0
Aug 01 00:18:50.687187 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:50.686913    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" event={"ID":"cbc10131-3c23-4348-82b2-b23c57e2de8b","Type":"ContainerDied","Data":"44230cc3b47a94246453beb8cc71a2f7d69f003d4567365724bfa442fd5cc95c"}
Aug 01 00:18:50.810667 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:50.810642    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:18:50.902209 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:50.902116    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc10131-3c23-4348-82b2-b23c57e2de8b-proxy-tls\") pod \"cbc10131-3c23-4348-82b2-b23c57e2de8b\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") "
Aug 01 00:18:50.902209 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:50.902155    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xmsr\" (UniqueName: \"kubernetes.io/projected/cbc10131-3c23-4348-82b2-b23c57e2de8b-kube-api-access-2xmsr\") pod \"cbc10131-3c23-4348-82b2-b23c57e2de8b\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") "
Aug 01 00:18:50.902209 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:50.902198    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbc10131-3c23-4348-82b2-b23c57e2de8b-cabundle-cert\") pod \"cbc10131-3c23-4348-82b2-b23c57e2de8b\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") "
Aug 01 00:18:50.902510 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:50.902226    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbc10131-3c23-4348-82b2-b23c57e2de8b-kserve-provision-location\") pod \"cbc10131-3c23-4348-82b2-b23c57e2de8b\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") "
Aug 01 00:18:50.902510 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:50.902352    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbc10131-3c23-4348-82b2-b23c57e2de8b-kube-rbac-proxy-sar-config\") pod \"cbc10131-3c23-4348-82b2-b23c57e2de8b\" (UID: \"cbc10131-3c23-4348-82b2-b23c57e2de8b\") "
Aug 01 00:18:50.902640 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:50.902613    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc10131-3c23-4348-82b2-b23c57e2de8b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cbc10131-3c23-4348-82b2-b23c57e2de8b" (UID: "cbc10131-3c23-4348-82b2-b23c57e2de8b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:18:50.902681 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:50.902591    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc10131-3c23-4348-82b2-b23c57e2de8b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cbc10131-3c23-4348-82b2-b23c57e2de8b" (UID: "cbc10131-3c23-4348-82b2-b23c57e2de8b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:18:50.902720 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:50.902705    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc10131-3c23-4348-82b2-b23c57e2de8b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cbc10131-3c23-4348-82b2-b23c57e2de8b" (UID: "cbc10131-3c23-4348-82b2-b23c57e2de8b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:18:50.904336 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:50.904312    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc10131-3c23-4348-82b2-b23c57e2de8b-kube-api-access-2xmsr" (OuterVolumeSpecName: "kube-api-access-2xmsr") pod "cbc10131-3c23-4348-82b2-b23c57e2de8b" (UID: "cbc10131-3c23-4348-82b2-b23c57e2de8b"). InnerVolumeSpecName "kube-api-access-2xmsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:18:50.904448 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:50.904395    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc10131-3c23-4348-82b2-b23c57e2de8b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cbc10131-3c23-4348-82b2-b23c57e2de8b" (UID: "cbc10131-3c23-4348-82b2-b23c57e2de8b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:18:51.003118 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:51.003088    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbc10131-3c23-4348-82b2-b23c57e2de8b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:18:51.003118 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:51.003113    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbc10131-3c23-4348-82b2-b23c57e2de8b-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:18:51.003118 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:51.003123    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xmsr\" (UniqueName: \"kubernetes.io/projected/cbc10131-3c23-4348-82b2-b23c57e2de8b-kube-api-access-2xmsr\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:18:51.003394 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:51.003134    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbc10131-3c23-4348-82b2-b23c57e2de8b-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:18:51.003394 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:51.003143    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbc10131-3c23-4348-82b2-b23c57e2de8b-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:18:51.221025 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:51.220939    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:51.654158 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:51.654121    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:18:51.692801 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:51.692771    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"
Aug 01 00:18:51.692801 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:51.692782    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp" event={"ID":"cbc10131-3c23-4348-82b2-b23c57e2de8b","Type":"ContainerDied","Data":"d5236a5b47a8dbd2e6505b259308cb09eaabb1e55f48534ba5faa9c95218feff"}
Aug 01 00:18:51.693304 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:51.692853    2546 scope.go:117] "RemoveContainer" containerID="a53f15edcbeec46a79beb1a920b932c56550a82d60ecc90ba5c5e3c1282161b1"
Aug 01 00:18:51.701699 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:51.701681    2546 scope.go:117] "RemoveContainer" containerID="44230cc3b47a94246453beb8cc71a2f7d69f003d4567365724bfa442fd5cc95c"
Aug 01 00:18:51.709193 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:51.709172    2546 scope.go:117] "RemoveContainer" containerID="fc7ab9691599989496958d2175dcc536029f011853cab016c920edeb00911333"
Aug 01 00:18:51.714781 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:51.714758    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"]
Aug 01 00:18:51.718761 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:51.718741    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7bffcff858-p4vzp"]
Aug 01 00:18:52.075293 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:18:52.075202    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" path="/var/lib/kubelet/pods/cbc10131-3c23-4348-82b2-b23c57e2de8b/volumes"
Aug 01 00:19:36.749534 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:36.749496    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"]
Aug 01 00:19:36.750072 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:36.749741    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kserve-container" containerID="cri-o://93cb5938f1b6c338d438584e7384979eeaebb0b3fba8a113dc5ba54e28356250" gracePeriod=30
Aug 01 00:19:36.750072 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:36.749768    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kube-rbac-proxy" containerID="cri-o://f73ce78f7ae2e9a4573d1bb0775cc013144f3376b78e30d70fcfd978cf1cdb8b" gracePeriod=30
Aug 01 00:19:36.769351 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:36.769327    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:19:36.877453 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:36.877424    2546 generic.go:356] "Generic (PLEG): container finished" podID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerID="f73ce78f7ae2e9a4573d1bb0775cc013144f3376b78e30d70fcfd978cf1cdb8b" exitCode=2
Aug 01 00:19:36.877589 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:36.877486    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" event={"ID":"6b62a662-f4c1-4293-b374-2a62c06c2d07","Type":"ContainerDied","Data":"f73ce78f7ae2e9a4573d1bb0775cc013144f3376b78e30d70fcfd978cf1cdb8b"}
Aug 01 00:19:38.069135 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:38.069092    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Aug 01 00:19:38.079243 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:38.079208    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Aug 01 00:19:38.808125 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:38.808090    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:19:41.593836 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.593811    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:19:41.712116 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.712028    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b62a662-f4c1-4293-b374-2a62c06c2d07-cabundle-cert\") pod \"6b62a662-f4c1-4293-b374-2a62c06c2d07\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") "
Aug 01 00:19:41.712116 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.712079    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdqpk\" (UniqueName: \"kubernetes.io/projected/6b62a662-f4c1-4293-b374-2a62c06c2d07-kube-api-access-gdqpk\") pod \"6b62a662-f4c1-4293-b374-2a62c06c2d07\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") "
Aug 01 00:19:41.712116 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.712111    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b62a662-f4c1-4293-b374-2a62c06c2d07-kube-rbac-proxy-sar-config\") pod \"6b62a662-f4c1-4293-b374-2a62c06c2d07\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") "
Aug 01 00:19:41.712440 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.712147    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b62a662-f4c1-4293-b374-2a62c06c2d07-proxy-tls\") pod \"6b62a662-f4c1-4293-b374-2a62c06c2d07\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") "
Aug 01 00:19:41.712440 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.712194    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b62a662-f4c1-4293-b374-2a62c06c2d07-kserve-provision-location\") pod \"6b62a662-f4c1-4293-b374-2a62c06c2d07\" (UID: \"6b62a662-f4c1-4293-b374-2a62c06c2d07\") "
Aug 01 00:19:41.712545 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.712495    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b62a662-f4c1-4293-b374-2a62c06c2d07-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6b62a662-f4c1-4293-b374-2a62c06c2d07" (UID: "6b62a662-f4c1-4293-b374-2a62c06c2d07"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:19:41.712545 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.712519    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b62a662-f4c1-4293-b374-2a62c06c2d07-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6b62a662-f4c1-4293-b374-2a62c06c2d07" (UID: "6b62a662-f4c1-4293-b374-2a62c06c2d07"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:19:41.712545 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.712533    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b62a662-f4c1-4293-b374-2a62c06c2d07-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6b62a662-f4c1-4293-b374-2a62c06c2d07" (UID: "6b62a662-f4c1-4293-b374-2a62c06c2d07"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:19:41.714238 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.714215    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b62a662-f4c1-4293-b374-2a62c06c2d07-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6b62a662-f4c1-4293-b374-2a62c06c2d07" (UID: "6b62a662-f4c1-4293-b374-2a62c06c2d07"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:19:41.714339 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.714243    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b62a662-f4c1-4293-b374-2a62c06c2d07-kube-api-access-gdqpk" (OuterVolumeSpecName: "kube-api-access-gdqpk") pod "6b62a662-f4c1-4293-b374-2a62c06c2d07" (UID: "6b62a662-f4c1-4293-b374-2a62c06c2d07"). InnerVolumeSpecName "kube-api-access-gdqpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:19:41.813041 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.813009    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b62a662-f4c1-4293-b374-2a62c06c2d07-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:19:41.813041 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.813036    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b62a662-f4c1-4293-b374-2a62c06c2d07-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:19:41.813041 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.813046    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdqpk\" (UniqueName: \"kubernetes.io/projected/6b62a662-f4c1-4293-b374-2a62c06c2d07-kube-api-access-gdqpk\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:19:41.813269 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.813055    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b62a662-f4c1-4293-b374-2a62c06c2d07-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:19:41.813269 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.813064    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b62a662-f4c1-4293-b374-2a62c06c2d07-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:19:41.870951 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.870898    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:19:41.899121 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.899092    2546 generic.go:356] "Generic (PLEG): container finished" podID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerID="93cb5938f1b6c338d438584e7384979eeaebb0b3fba8a113dc5ba54e28356250" exitCode=0
Aug 01 00:19:41.899297 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.899139    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" event={"ID":"6b62a662-f4c1-4293-b374-2a62c06c2d07","Type":"ContainerDied","Data":"93cb5938f1b6c338d438584e7384979eeaebb0b3fba8a113dc5ba54e28356250"}
Aug 01 00:19:41.899297 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.899161    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"
Aug 01 00:19:41.899297 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.899184    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh" event={"ID":"6b62a662-f4c1-4293-b374-2a62c06c2d07","Type":"ContainerDied","Data":"b22cde366106176a7054c137683ea1ed661cbb6f355735113ab7542909655125"}
Aug 01 00:19:41.899297 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.899211    2546 scope.go:117] "RemoveContainer" containerID="f73ce78f7ae2e9a4573d1bb0775cc013144f3376b78e30d70fcfd978cf1cdb8b"
Aug 01 00:19:41.902898 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.902877    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"]
Aug 01 00:19:41.903209 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903195    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kube-rbac-proxy"
Aug 01 00:19:41.903209 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903210    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kube-rbac-proxy"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903216    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kube-rbac-proxy"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903222    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kube-rbac-proxy"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903233    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="storage-initializer"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903242    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="storage-initializer"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903267    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e711c08-3106-4f85-8169-38a1f5be2add" containerName="kserve-container"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903273    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e711c08-3106-4f85-8169-38a1f5be2add" containerName="kserve-container"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903279    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kserve-container"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903285    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kserve-container"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903294    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kserve-container"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903299    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kserve-container"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903313    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="storage-initializer"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903321    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="storage-initializer"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903330    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kserve-container"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903335    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kserve-container"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903344    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kube-rbac-proxy"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903350    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kube-rbac-proxy"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903356    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e711c08-3106-4f85-8169-38a1f5be2add" containerName="storage-initializer"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903361    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e711c08-3106-4f85-8169-38a1f5be2add" containerName="storage-initializer"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903369    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e711c08-3106-4f85-8169-38a1f5be2add" containerName="kube-rbac-proxy"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903374    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e711c08-3106-4f85-8169-38a1f5be2add" containerName="kube-rbac-proxy"
Aug 01 00:19:41.903372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903381    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="storage-initializer"
Aug 01 00:19:41.904459 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903389    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="storage-initializer"
Aug 01 00:19:41.904459 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903442    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kube-rbac-proxy"
Aug 01 00:19:41.904459 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903450    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kserve-container"
Aug 01 00:19:41.904459 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903456    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc10131-3c23-4348-82b2-b23c57e2de8b" containerName="kube-rbac-proxy"
Aug 01 00:19:41.904459 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903463    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e711c08-3106-4f85-8169-38a1f5be2add" containerName="kserve-container"
Aug 01 00:19:41.904459 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903472    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d5b1c45-5e2d-4475-8d59-214ab5c1b7b2" containerName="kserve-container"
Aug 01 00:19:41.904459 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903478    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e711c08-3106-4f85-8169-38a1f5be2add" containerName="kube-rbac-proxy"
Aug 01 00:19:41.904459 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903485    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kserve-container"
Aug 01 00:19:41.904459 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.903490    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" containerName="kube-rbac-proxy"
Aug 01 00:19:41.907853 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.907837    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:41.908264 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.908140    2546 scope.go:117] "RemoveContainer" containerID="93cb5938f1b6c338d438584e7384979eeaebb0b3fba8a113dc5ba54e28356250"
Aug 01 00:19:41.910346 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.910321    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Aug 01 00:19:41.910346 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.910343    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:19:41.910499 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.910403    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:19:41.910499 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.910493    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Aug 01 00:19:41.910602 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.910501    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:19:41.910695 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.910682    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:19:41.910751 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.910699    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:19:41.921034 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.918937    2546 scope.go:117] "RemoveContainer" containerID="a8be7767f2aee512d2e7132580be9020859a14622022979d4e181f20995c8d92"
Aug 01 00:19:41.922606 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.922561    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"]
Aug 01 00:19:41.931578 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.931553    2546 scope.go:117] "RemoveContainer" containerID="f73ce78f7ae2e9a4573d1bb0775cc013144f3376b78e30d70fcfd978cf1cdb8b"
Aug 01 00:19:41.932015 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:19:41.931914    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73ce78f7ae2e9a4573d1bb0775cc013144f3376b78e30d70fcfd978cf1cdb8b\": container with ID starting with f73ce78f7ae2e9a4573d1bb0775cc013144f3376b78e30d70fcfd978cf1cdb8b not found: ID does not exist" containerID="f73ce78f7ae2e9a4573d1bb0775cc013144f3376b78e30d70fcfd978cf1cdb8b"
Aug 01 00:19:41.932188 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.932012    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73ce78f7ae2e9a4573d1bb0775cc013144f3376b78e30d70fcfd978cf1cdb8b"} err="failed to get container status \"f73ce78f7ae2e9a4573d1bb0775cc013144f3376b78e30d70fcfd978cf1cdb8b\": rpc error: code = NotFound desc = could not find container \"f73ce78f7ae2e9a4573d1bb0775cc013144f3376b78e30d70fcfd978cf1cdb8b\": container with ID starting with f73ce78f7ae2e9a4573d1bb0775cc013144f3376b78e30d70fcfd978cf1cdb8b not found: ID does not exist"
Aug 01 00:19:41.932188 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.932039    2546 scope.go:117] "RemoveContainer" containerID="93cb5938f1b6c338d438584e7384979eeaebb0b3fba8a113dc5ba54e28356250"
Aug 01 00:19:41.932824 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:19:41.932567    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93cb5938f1b6c338d438584e7384979eeaebb0b3fba8a113dc5ba54e28356250\": container with ID starting with 93cb5938f1b6c338d438584e7384979eeaebb0b3fba8a113dc5ba54e28356250 not found: ID does not exist" containerID="93cb5938f1b6c338d438584e7384979eeaebb0b3fba8a113dc5ba54e28356250"
Aug 01 00:19:41.932824 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.932596    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93cb5938f1b6c338d438584e7384979eeaebb0b3fba8a113dc5ba54e28356250"} err="failed to get container status \"93cb5938f1b6c338d438584e7384979eeaebb0b3fba8a113dc5ba54e28356250\": rpc error: code = NotFound desc = could not find container \"93cb5938f1b6c338d438584e7384979eeaebb0b3fba8a113dc5ba54e28356250\": container with ID starting with 93cb5938f1b6c338d438584e7384979eeaebb0b3fba8a113dc5ba54e28356250 not found: ID does not exist"
Aug 01 00:19:41.932824 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.932618    2546 scope.go:117] "RemoveContainer" containerID="a8be7767f2aee512d2e7132580be9020859a14622022979d4e181f20995c8d92"
Aug 01 00:19:41.932967 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:19:41.932907    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8be7767f2aee512d2e7132580be9020859a14622022979d4e181f20995c8d92\": container with ID starting with a8be7767f2aee512d2e7132580be9020859a14622022979d4e181f20995c8d92 not found: ID does not exist" containerID="a8be7767f2aee512d2e7132580be9020859a14622022979d4e181f20995c8d92"
Aug 01 00:19:41.932967 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.932935    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8be7767f2aee512d2e7132580be9020859a14622022979d4e181f20995c8d92"} err="failed to get container status \"a8be7767f2aee512d2e7132580be9020859a14622022979d4e181f20995c8d92\": rpc error: code = NotFound desc = could not find container \"a8be7767f2aee512d2e7132580be9020859a14622022979d4e181f20995c8d92\": container with ID starting with a8be7767f2aee512d2e7132580be9020859a14622022979d4e181f20995c8d92 not found: ID does not exist"
Aug 01 00:19:41.942777 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.942757    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"]
Aug 01 00:19:41.947626 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.947607    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"]
Aug 01 00:19:41.947704 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.947686    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:41.949844 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.949829    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Aug 01 00:19:41.955308 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.955283    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:19:41.957830 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.957804    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"]
Aug 01 00:19:41.959326 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:41.959307    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7b47596d4d-4vclh"]
Aug 01 00:19:42.014546 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.014479    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:19:42.015599 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.015578    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ab31c82-b82d-4675-ac1e-0b69877cc222-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-8589bf8f95-hl4kz\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.015657 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.015608    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42l5x\" (UniqueName: \"kubernetes.io/projected/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kube-api-access-42l5x\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.015657 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.015646    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ab31c82-b82d-4675-ac1e-0b69877cc222-cabundle-cert\") pod \"isvc-canary-rollback-predictor-8589bf8f95-hl4kz\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.015726 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.015689    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.015726 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.015710    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.015790 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.015753    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.015790 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.015773    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qxqs\" (UniqueName: \"kubernetes.io/projected/8ab31c82-b82d-4675-ac1e-0b69877cc222-kube-api-access-4qxqs\") pod \"isvc-canary-rollback-predictor-8589bf8f95-hl4kz\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.015855 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.015814    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ab31c82-b82d-4675-ac1e-0b69877cc222-proxy-tls\") pod \"isvc-canary-rollback-predictor-8589bf8f95-hl4kz\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.015889 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.015853    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ab31c82-b82d-4675-ac1e-0b69877cc222-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-8589bf8f95-hl4kz\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.015923 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.015889    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.075134 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.075104    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b62a662-f4c1-4293-b374-2a62c06c2d07" path="/var/lib/kubelet/pods/6b62a662-f4c1-4293-b374-2a62c06c2d07/volumes"
Aug 01 00:19:42.116230 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.116204    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ab31c82-b82d-4675-ac1e-0b69877cc222-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-8589bf8f95-hl4kz\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.116374 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.116241    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42l5x\" (UniqueName: \"kubernetes.io/projected/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kube-api-access-42l5x\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.116374 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.116292    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ab31c82-b82d-4675-ac1e-0b69877cc222-cabundle-cert\") pod \"isvc-canary-rollback-predictor-8589bf8f95-hl4kz\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.116374 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.116317    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.116374 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.116337    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.116374 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.116373    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.116623 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.116402    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qxqs\" (UniqueName: \"kubernetes.io/projected/8ab31c82-b82d-4675-ac1e-0b69877cc222-kube-api-access-4qxqs\") pod \"isvc-canary-rollback-predictor-8589bf8f95-hl4kz\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.116623 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.116435    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ab31c82-b82d-4675-ac1e-0b69877cc222-proxy-tls\") pod \"isvc-canary-rollback-predictor-8589bf8f95-hl4kz\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.116623 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.116505    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ab31c82-b82d-4675-ac1e-0b69877cc222-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-8589bf8f95-hl4kz\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.116623 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.116547    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.116838 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.116618    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ab31c82-b82d-4675-ac1e-0b69877cc222-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-8589bf8f95-hl4kz\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.116838 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.116735    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.117100 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.117070    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ab31c82-b82d-4675-ac1e-0b69877cc222-cabundle-cert\") pod \"isvc-canary-rollback-predictor-8589bf8f95-hl4kz\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.117190 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.117142    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.117285 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.117240    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ab31c82-b82d-4675-ac1e-0b69877cc222-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-8589bf8f95-hl4kz\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.117364 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.117338    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.118930 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.118903    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.119023 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.119005    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ab31c82-b82d-4675-ac1e-0b69877cc222-proxy-tls\") pod \"isvc-canary-rollback-predictor-8589bf8f95-hl4kz\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.124131 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.124111    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42l5x\" (UniqueName: \"kubernetes.io/projected/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kube-api-access-42l5x\") pod \"isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.126863 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.126845    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qxqs\" (UniqueName: \"kubernetes.io/projected/8ab31c82-b82d-4675-ac1e-0b69877cc222-kube-api-access-4qxqs\") pod \"isvc-canary-rollback-predictor-8589bf8f95-hl4kz\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.221883 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.221853    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:42.258228 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.258190    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:42.367639 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.367276    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"]
Aug 01 00:19:42.369370 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:19:42.369341    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab31c82_b82d_4675_ac1e_0b69877cc222.slice/crio-b9f209e542b71abf88a0b1f3f6a183b6a8b3634f149fad9835dd20d4ab9f7992 WatchSource:0}: Error finding container b9f209e542b71abf88a0b1f3f6a183b6a8b3634f149fad9835dd20d4ab9f7992: Status 404 returned error can't find the container with id b9f209e542b71abf88a0b1f3f6a183b6a8b3634f149fad9835dd20d4ab9f7992
Aug 01 00:19:42.410431 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.410408    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"]
Aug 01 00:19:42.412843 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:19:42.412818    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0b6a9f_9a03_4913_b095_4b95fea4b0d5.slice/crio-3cf866d938e817690b034bb202cdd38e4995ea3e0d17709cc63dcd3b3842e68a WatchSource:0}: Error finding container 3cf866d938e817690b034bb202cdd38e4995ea3e0d17709cc63dcd3b3842e68a: Status 404 returned error can't find the container with id 3cf866d938e817690b034bb202cdd38e4995ea3e0d17709cc63dcd3b3842e68a
Aug 01 00:19:42.906735 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.906688    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" event={"ID":"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5","Type":"ContainerStarted","Data":"59b29c0ebb985937755379572747222fee2be4e1d6699b9001895954ea9cee05"}
Aug 01 00:19:42.907200 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.906766    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" event={"ID":"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5","Type":"ContainerStarted","Data":"3cf866d938e817690b034bb202cdd38e4995ea3e0d17709cc63dcd3b3842e68a"}
Aug 01 00:19:42.908387 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.908362    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" event={"ID":"8ab31c82-b82d-4675-ac1e-0b69877cc222","Type":"ContainerStarted","Data":"f7851e944d8569554debfcdb7eea71a8e5075b970951d16e6f2e1cf5479fff53"}
Aug 01 00:19:42.908498 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:42.908409    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" event={"ID":"8ab31c82-b82d-4675-ac1e-0b69877cc222","Type":"ContainerStarted","Data":"b9f209e542b71abf88a0b1f3f6a183b6a8b3634f149fad9835dd20d4ab9f7992"}
Aug 01 00:19:46.928313 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:46.928280    2546 generic.go:356] "Generic (PLEG): container finished" podID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerID="59b29c0ebb985937755379572747222fee2be4e1d6699b9001895954ea9cee05" exitCode=0
Aug 01 00:19:46.928723 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:46.928347    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" event={"ID":"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5","Type":"ContainerDied","Data":"59b29c0ebb985937755379572747222fee2be4e1d6699b9001895954ea9cee05"}
Aug 01 00:19:46.929938 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:46.929916    2546 generic.go:356] "Generic (PLEG): container finished" podID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerID="f7851e944d8569554debfcdb7eea71a8e5075b970951d16e6f2e1cf5479fff53" exitCode=0
Aug 01 00:19:46.930038 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:46.929994    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" event={"ID":"8ab31c82-b82d-4675-ac1e-0b69877cc222","Type":"ContainerDied","Data":"f7851e944d8569554debfcdb7eea71a8e5075b970951d16e6f2e1cf5479fff53"}
Aug 01 00:19:47.939530 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:47.939495    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" event={"ID":"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5","Type":"ContainerStarted","Data":"f79597b57af35ec53b40660495e51234ed1e0f9e2615bf61162930323f9a3122"}
Aug 01 00:19:47.939945 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:47.939565    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" event={"ID":"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5","Type":"ContainerStarted","Data":"d50695640920c7c72ae447a35597e926095e55cd99ff36aa3319263f660a75cc"}
Aug 01 00:19:47.939945 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:47.939873    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:47.940081 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:47.940040    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:47.941556 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:47.941527    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Aug 01 00:19:47.941678 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:47.941660    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" event={"ID":"8ab31c82-b82d-4675-ac1e-0b69877cc222","Type":"ContainerStarted","Data":"e4dfae5f2889c6ce6240644308bfe756948a370f74a5fed0b606c1692fd1b9ae"}
Aug 01 00:19:47.941732 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:47.941707    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" event={"ID":"8ab31c82-b82d-4675-ac1e-0b69877cc222","Type":"ContainerStarted","Data":"0213dd4705349d29006a74966f0e1172b29282644cd2d419e5a392dcabff188d"}
Aug 01 00:19:47.941907 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:47.941893    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:47.957919 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:47.957869    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" podStartSLOduration=3.144026487 podStartE2EDuration="6.957851114s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.813824627s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:19:47.957483778 +0000 UTC m=+2018.416946591" watchObservedRunningTime="2026-08-01 00:19:47.957851114 +0000 UTC m=+2018.417313928"
Aug 01 00:19:47.975536 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:47.975482    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podStartSLOduration=3.250167751 podStartE2EDuration="6.975467767s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.725300016s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:19:47.974908614 +0000 UTC m=+2018.434371418" watchObservedRunningTime="2026-08-01 00:19:47.975467767 +0000 UTC m=+2018.434930580"
Aug 01 00:19:48.946866 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:48.946835    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:48.947271 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:48.946833    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Aug 01 00:19:48.947958 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:48.947929    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Aug 01 00:19:49.950576 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:49.950540    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Aug 01 00:19:53.951285 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:53.951234    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:19:53.951858 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:53.951830    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Aug 01 00:19:54.955471 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:54.955443    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:19:54.956088 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:19:54.956058    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Aug 01 00:20:03.951813 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:03.951772    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Aug 01 00:20:04.956214 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:04.956175    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Aug 01 00:20:13.951953 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:13.951912    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Aug 01 00:20:14.956669 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:14.956627    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Aug 01 00:20:21.373832 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:21.373798    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:20:23.951906 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:23.951867    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Aug 01 00:20:24.956208 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:24.956168    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Aug 01 00:20:26.515170 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:26.515134    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:20:26.697117 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:26.697086    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:20:33.951819 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:33.951779    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Aug 01 00:20:34.956804 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:34.956762    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Aug 01 00:20:43.952496 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:43.952452    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Aug 01 00:20:44.956165 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:44.956124    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Aug 01 00:20:53.952509 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:53.952466    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Aug 01 00:20:54.956909 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:54.956873    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Aug 01 00:20:59.292966 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:20:59.292931    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:02.075242 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:02.075214    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:21:04.957164 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:04.957131    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:21:07.373661 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:07.373633    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:11.926340 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:11.926304    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"]
Aug 01 00:21:11.926736 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:11.926587    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kserve-container" containerID="cri-o://d50695640920c7c72ae447a35597e926095e55cd99ff36aa3319263f660a75cc" gracePeriod=30
Aug 01 00:21:11.926813 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:11.926733    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kube-rbac-proxy" containerID="cri-o://f79597b57af35ec53b40660495e51234ed1e0f9e2615bf61162930323f9a3122" gracePeriod=30
Aug 01 00:21:11.956192 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:11.956171    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:12.071332 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:12.071293    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Aug 01 00:21:12.289822 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:12.289732    2546 generic.go:356] "Generic (PLEG): container finished" podID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerID="f79597b57af35ec53b40660495e51234ed1e0f9e2615bf61162930323f9a3122" exitCode=2
Aug 01 00:21:12.289822 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:12.289784    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" event={"ID":"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5","Type":"ContainerDied","Data":"f79597b57af35ec53b40660495e51234ed1e0f9e2615bf61162930323f9a3122"}
Aug 01 00:21:12.600629 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:12.600542    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:12.782834 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:12.782801    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:13.947624 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:13.947571    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Aug 01 00:21:16.275623 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.275600    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:21:16.313393 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.313357    2546 generic.go:356] "Generic (PLEG): container finished" podID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerID="d50695640920c7c72ae447a35597e926095e55cd99ff36aa3319263f660a75cc" exitCode=0
Aug 01 00:21:16.313546 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.313411    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"
Aug 01 00:21:16.313546 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.313432    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" event={"ID":"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5","Type":"ContainerDied","Data":"d50695640920c7c72ae447a35597e926095e55cd99ff36aa3319263f660a75cc"}
Aug 01 00:21:16.313546 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.313498    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9" event={"ID":"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5","Type":"ContainerDied","Data":"3cf866d938e817690b034bb202cdd38e4995ea3e0d17709cc63dcd3b3842e68a"}
Aug 01 00:21:16.313546 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.313529    2546 scope.go:117] "RemoveContainer" containerID="f79597b57af35ec53b40660495e51234ed1e0f9e2615bf61162930323f9a3122"
Aug 01 00:21:16.322708 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.322691    2546 scope.go:117] "RemoveContainer" containerID="d50695640920c7c72ae447a35597e926095e55cd99ff36aa3319263f660a75cc"
Aug 01 00:21:16.330552 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.330531    2546 scope.go:117] "RemoveContainer" containerID="59b29c0ebb985937755379572747222fee2be4e1d6699b9001895954ea9cee05"
Aug 01 00:21:16.338050 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.338032    2546 scope.go:117] "RemoveContainer" containerID="f79597b57af35ec53b40660495e51234ed1e0f9e2615bf61162930323f9a3122"
Aug 01 00:21:16.338346 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:21:16.338325    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f79597b57af35ec53b40660495e51234ed1e0f9e2615bf61162930323f9a3122\": container with ID starting with f79597b57af35ec53b40660495e51234ed1e0f9e2615bf61162930323f9a3122 not found: ID does not exist" containerID="f79597b57af35ec53b40660495e51234ed1e0f9e2615bf61162930323f9a3122"
Aug 01 00:21:16.338420 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.338355    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f79597b57af35ec53b40660495e51234ed1e0f9e2615bf61162930323f9a3122"} err="failed to get container status \"f79597b57af35ec53b40660495e51234ed1e0f9e2615bf61162930323f9a3122\": rpc error: code = NotFound desc = could not find container \"f79597b57af35ec53b40660495e51234ed1e0f9e2615bf61162930323f9a3122\": container with ID starting with f79597b57af35ec53b40660495e51234ed1e0f9e2615bf61162930323f9a3122 not found: ID does not exist"
Aug 01 00:21:16.338420 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.338374    2546 scope.go:117] "RemoveContainer" containerID="d50695640920c7c72ae447a35597e926095e55cd99ff36aa3319263f660a75cc"
Aug 01 00:21:16.338616 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:21:16.338600    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50695640920c7c72ae447a35597e926095e55cd99ff36aa3319263f660a75cc\": container with ID starting with d50695640920c7c72ae447a35597e926095e55cd99ff36aa3319263f660a75cc not found: ID does not exist" containerID="d50695640920c7c72ae447a35597e926095e55cd99ff36aa3319263f660a75cc"
Aug 01 00:21:16.338658 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.338622    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50695640920c7c72ae447a35597e926095e55cd99ff36aa3319263f660a75cc"} err="failed to get container status \"d50695640920c7c72ae447a35597e926095e55cd99ff36aa3319263f660a75cc\": rpc error: code = NotFound desc = could not find container \"d50695640920c7c72ae447a35597e926095e55cd99ff36aa3319263f660a75cc\": container with ID starting with d50695640920c7c72ae447a35597e926095e55cd99ff36aa3319263f660a75cc not found: ID does not exist"
Aug 01 00:21:16.338658 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.338637    2546 scope.go:117] "RemoveContainer" containerID="59b29c0ebb985937755379572747222fee2be4e1d6699b9001895954ea9cee05"
Aug 01 00:21:16.338841 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:21:16.338824    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b29c0ebb985937755379572747222fee2be4e1d6699b9001895954ea9cee05\": container with ID starting with 59b29c0ebb985937755379572747222fee2be4e1d6699b9001895954ea9cee05 not found: ID does not exist" containerID="59b29c0ebb985937755379572747222fee2be4e1d6699b9001895954ea9cee05"
Aug 01 00:21:16.338885 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.338845    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b29c0ebb985937755379572747222fee2be4e1d6699b9001895954ea9cee05"} err="failed to get container status \"59b29c0ebb985937755379572747222fee2be4e1d6699b9001895954ea9cee05\": rpc error: code = NotFound desc = could not find container \"59b29c0ebb985937755379572747222fee2be4e1d6699b9001895954ea9cee05\": container with ID starting with 59b29c0ebb985937755379572747222fee2be4e1d6699b9001895954ea9cee05 not found: ID does not exist"
Aug 01 00:21:16.391601 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.391516    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-proxy-tls\") pod \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") "
Aug 01 00:21:16.391601 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.391555    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kube-rbac-proxy-sar-config\") pod \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") "
Aug 01 00:21:16.391601 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.391579    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42l5x\" (UniqueName: \"kubernetes.io/projected/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kube-api-access-42l5x\") pod \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") "
Aug 01 00:21:16.391601 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.391595    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kserve-provision-location\") pod \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") "
Aug 01 00:21:16.391947 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.391660    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-cabundle-cert\") pod \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\" (UID: \"3a0b6a9f-9a03-4913-b095-4b95fea4b0d5\") "
Aug 01 00:21:16.391989 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.391952    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" (UID: "3a0b6a9f-9a03-4913-b095-4b95fea4b0d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:21:16.392024 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.391985    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" (UID: "3a0b6a9f-9a03-4913-b095-4b95fea4b0d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:21:16.392062 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.392025    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" (UID: "3a0b6a9f-9a03-4913-b095-4b95fea4b0d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:21:16.393712 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.393688    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" (UID: "3a0b6a9f-9a03-4913-b095-4b95fea4b0d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:21:16.393803 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.393747    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kube-api-access-42l5x" (OuterVolumeSpecName: "kube-api-access-42l5x") pod "3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" (UID: "3a0b6a9f-9a03-4913-b095-4b95fea4b0d5"). InnerVolumeSpecName "kube-api-access-42l5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:21:16.493000 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.492953    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:21:16.493000 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.492996    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:21:16.493168 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.493009    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:21:16.493168 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.493041    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42l5x\" (UniqueName: \"kubernetes.io/projected/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kube-api-access-42l5x\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:21:16.493168 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.493053    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:21:16.635938 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.635906    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"]
Aug 01 00:21:16.641516 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:16.641490    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b7965d9d-ftgq9"]
Aug 01 00:21:18.079081 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:18.079044    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" path="/var/lib/kubelet/pods/3a0b6a9f-9a03-4913-b095-4b95fea4b0d5/volumes"
Aug 01 00:21:53.670803 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:53.670772    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:58.807546 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.807514    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:58.840865 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.840837    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"]
Aug 01 00:21:58.841191 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.841179    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="storage-initializer"
Aug 01 00:21:58.841241 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.841192    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="storage-initializer"
Aug 01 00:21:58.841241 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.841202    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kserve-container"
Aug 01 00:21:58.841241 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.841208    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kserve-container"
Aug 01 00:21:58.841241 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.841218    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kube-rbac-proxy"
Aug 01 00:21:58.841241 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.841224    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kube-rbac-proxy"
Aug 01 00:21:58.841411 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.841292    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kserve-container"
Aug 01 00:21:58.841411 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.841301    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a0b6a9f-9a03-4913-b095-4b95fea4b0d5" containerName="kube-rbac-proxy"
Aug 01 00:21:58.844342 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.844327    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:58.846790 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.846768    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Aug 01 00:21:58.846790 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.846780    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Aug 01 00:21:58.853705 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.853683    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"]
Aug 01 00:21:58.937590 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.937562    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjmmg\" (UniqueName: \"kubernetes.io/projected/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kube-api-access-cjmmg\") pod \"isvc-triton-predictor-5d7db58676-jfc6x\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:58.937761 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.937604    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f882bb7-a819-4f26-ad6f-fd12d7948a84-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-jfc6x\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:58.937761 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.937627    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-jfc6x\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:58.937761 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.937658    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-jfc6x\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:58.937761 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.937685    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f882bb7-a819-4f26-ad6f-fd12d7948a84-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-jfc6x\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:58.947604 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:58.947576    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:21:59.038616 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:59.038589    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-jfc6x\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:59.038756 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:59.038622    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-jfc6x\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:59.038756 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:59.038639    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f882bb7-a819-4f26-ad6f-fd12d7948a84-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-jfc6x\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:59.038756 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:59.038722    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjmmg\" (UniqueName: \"kubernetes.io/projected/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kube-api-access-cjmmg\") pod \"isvc-triton-predictor-5d7db58676-jfc6x\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:59.038942 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:59.038770    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f882bb7-a819-4f26-ad6f-fd12d7948a84-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-jfc6x\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:59.039035 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:59.039014    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-jfc6x\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:59.039358 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:59.039339    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-jfc6x\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:59.039407 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:59.039342    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f882bb7-a819-4f26-ad6f-fd12d7948a84-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-jfc6x\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:59.041388 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:59.041369    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f882bb7-a819-4f26-ad6f-fd12d7948a84-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-jfc6x\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:59.047412 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:59.047387    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjmmg\" (UniqueName: \"kubernetes.io/projected/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kube-api-access-cjmmg\") pod \"isvc-triton-predictor-5d7db58676-jfc6x\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:59.154283 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:59.154245    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:21:59.287157 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:59.287134    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"]
Aug 01 00:21:59.289544 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:21:59.289513    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f882bb7_a819_4f26_ad6f_fd12d7948a84.slice/crio-452860ee54f48efc9ef07586aab6dc378f6fc90f334f061facebed36e37934fe WatchSource:0}: Error finding container 452860ee54f48efc9ef07586aab6dc378f6fc90f334f061facebed36e37934fe: Status 404 returned error can't find the container with id 452860ee54f48efc9ef07586aab6dc378f6fc90f334f061facebed36e37934fe
Aug 01 00:21:59.483883 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:59.483805    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x" event={"ID":"5f882bb7-a819-4f26-ad6f-fd12d7948a84","Type":"ContainerStarted","Data":"d51b59d51bc8c04b67d485502034164a0b069b4c457f96a7259370428d998dd6"}
Aug 01 00:21:59.483883 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:21:59.483862    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x" event={"ID":"5f882bb7-a819-4f26-ad6f-fd12d7948a84","Type":"ContainerStarted","Data":"452860ee54f48efc9ef07586aab6dc378f6fc90f334f061facebed36e37934fe"}
Aug 01 00:22:03.500210 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:03.500178    2546 generic.go:356] "Generic (PLEG): container finished" podID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" containerID="d51b59d51bc8c04b67d485502034164a0b069b4c457f96a7259370428d998dd6" exitCode=0
Aug 01 00:22:03.500210 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:03.500215    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x" event={"ID":"5f882bb7-a819-4f26-ad6f-fd12d7948a84","Type":"ContainerDied","Data":"d51b59d51bc8c04b67d485502034164a0b069b4c457f96a7259370428d998dd6"}
Aug 01 00:22:03.501338 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:03.501320    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:22:42.069326 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:42.069282    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"]
Aug 01 00:22:42.069890 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:42.069633    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container" containerID="cri-o://0213dd4705349d29006a74966f0e1172b29282644cd2d419e5a392dcabff188d" gracePeriod=30
Aug 01 00:22:42.069890 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:42.069815    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kube-rbac-proxy" containerID="cri-o://e4dfae5f2889c6ce6240644308bfe756948a370f74a5fed0b606c1692fd1b9ae" gracePeriod=30
Aug 01 00:22:42.091938 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:42.091909    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:42.738612 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:42.738574    2546 generic.go:356] "Generic (PLEG): container finished" podID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerID="e4dfae5f2889c6ce6240644308bfe756948a370f74a5fed0b606c1692fd1b9ae" exitCode=2
Aug 01 00:22:42.738801 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:42.738651    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" event={"ID":"8ab31c82-b82d-4675-ac1e-0b69877cc222","Type":"ContainerDied","Data":"e4dfae5f2889c6ce6240644308bfe756948a370f74a5fed0b606c1692fd1b9ae"}
Aug 01 00:22:44.951476 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:44.951397    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Aug 01 00:22:44.956176 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:44.956119    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Aug 01 00:22:47.239375 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:47.239334    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:47.306916 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:47.306882    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:47.369724 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:47.369688    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:47.619740 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:47.619696    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:49.951244 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:49.951198    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Aug 01 00:22:54.950971 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:54.950931    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Aug 01 00:22:54.951605 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:54.951061    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:22:54.956464 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:54.956428    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Aug 01 00:22:56.197661 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:56.197624    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:56.249030 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:56.249001    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:22:59.951115 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:22:59.951057    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Aug 01 00:23:04.951579 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:04.951530    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Aug 01 00:23:04.956873 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:04.956842    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Aug 01 00:23:04.957008 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:04.956978    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:23:09.951331 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:09.951289    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Aug 01 00:23:14.951689 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:14.951646    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Aug 01 00:23:14.956936 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:14.956903    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Aug 01 00:23:15.874212 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:15.874180    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:23:15.912119 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:15.912036    2546 generic.go:356] "Generic (PLEG): container finished" podID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerID="0213dd4705349d29006a74966f0e1172b29282644cd2d419e5a392dcabff188d" exitCode=0
Aug 01 00:23:15.912119 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:15.912087    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" event={"ID":"8ab31c82-b82d-4675-ac1e-0b69877cc222","Type":"ContainerDied","Data":"0213dd4705349d29006a74966f0e1172b29282644cd2d419e5a392dcabff188d"}
Aug 01 00:23:15.912119 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:15.912114    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"
Aug 01 00:23:15.912368 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:15.912163    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz" event={"ID":"8ab31c82-b82d-4675-ac1e-0b69877cc222","Type":"ContainerDied","Data":"b9f209e542b71abf88a0b1f3f6a183b6a8b3634f149fad9835dd20d4ab9f7992"}
Aug 01 00:23:15.912368 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:15.912190    2546 scope.go:117] "RemoveContainer" containerID="e4dfae5f2889c6ce6240644308bfe756948a370f74a5fed0b606c1692fd1b9ae"
Aug 01 00:23:15.936481 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:15.936457    2546 scope.go:117] "RemoveContainer" containerID="0213dd4705349d29006a74966f0e1172b29282644cd2d419e5a392dcabff188d"
Aug 01 00:23:15.999200 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:15.999163    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ab31c82-b82d-4675-ac1e-0b69877cc222-proxy-tls\") pod \"8ab31c82-b82d-4675-ac1e-0b69877cc222\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") "
Aug 01 00:23:15.999687 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:15.999274    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ab31c82-b82d-4675-ac1e-0b69877cc222-kube-rbac-proxy-sar-config\") pod \"8ab31c82-b82d-4675-ac1e-0b69877cc222\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") "
Aug 01 00:23:15.999687 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:15.999299    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qxqs\" (UniqueName: \"kubernetes.io/projected/8ab31c82-b82d-4675-ac1e-0b69877cc222-kube-api-access-4qxqs\") pod \"8ab31c82-b82d-4675-ac1e-0b69877cc222\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") "
Aug 01 00:23:15.999687 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:15.999346    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ab31c82-b82d-4675-ac1e-0b69877cc222-cabundle-cert\") pod \"8ab31c82-b82d-4675-ac1e-0b69877cc222\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") "
Aug 01 00:23:15.999687 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:15.999379    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ab31c82-b82d-4675-ac1e-0b69877cc222-kserve-provision-location\") pod \"8ab31c82-b82d-4675-ac1e-0b69877cc222\" (UID: \"8ab31c82-b82d-4675-ac1e-0b69877cc222\") "
Aug 01 00:23:15.999687 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:15.999608    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab31c82-b82d-4675-ac1e-0b69877cc222-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ab31c82-b82d-4675-ac1e-0b69877cc222" (UID: "8ab31c82-b82d-4675-ac1e-0b69877cc222"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:23:15.999909 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:15.999750    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab31c82-b82d-4675-ac1e-0b69877cc222-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ab31c82-b82d-4675-ac1e-0b69877cc222" (UID: "8ab31c82-b82d-4675-ac1e-0b69877cc222"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:23:15.999909 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:15.999750    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab31c82-b82d-4675-ac1e-0b69877cc222-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ab31c82-b82d-4675-ac1e-0b69877cc222" (UID: "8ab31c82-b82d-4675-ac1e-0b69877cc222"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:23:16.001404 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.001382    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab31c82-b82d-4675-ac1e-0b69877cc222-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ab31c82-b82d-4675-ac1e-0b69877cc222" (UID: "8ab31c82-b82d-4675-ac1e-0b69877cc222"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:23:16.001404 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.001388    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab31c82-b82d-4675-ac1e-0b69877cc222-kube-api-access-4qxqs" (OuterVolumeSpecName: "kube-api-access-4qxqs") pod "8ab31c82-b82d-4675-ac1e-0b69877cc222" (UID: "8ab31c82-b82d-4675-ac1e-0b69877cc222"). InnerVolumeSpecName "kube-api-access-4qxqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:23:16.048134 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.048098    2546 scope.go:117] "RemoveContainer" containerID="f7851e944d8569554debfcdb7eea71a8e5075b970951d16e6f2e1cf5479fff53"
Aug 01 00:23:16.100126 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.100090    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ab31c82-b82d-4675-ac1e-0b69877cc222-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:23:16.100126 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.100121    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qxqs\" (UniqueName: \"kubernetes.io/projected/8ab31c82-b82d-4675-ac1e-0b69877cc222-kube-api-access-4qxqs\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:23:16.100126 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.100131    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ab31c82-b82d-4675-ac1e-0b69877cc222-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:23:16.100396 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.100141    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ab31c82-b82d-4675-ac1e-0b69877cc222-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:23:16.100396 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.100152    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ab31c82-b82d-4675-ac1e-0b69877cc222-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:23:16.230401 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.230319    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"]
Aug 01 00:23:16.233967 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.233939    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-8589bf8f95-hl4kz"]
Aug 01 00:23:16.766983 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.766695    2546 scope.go:117] "RemoveContainer" containerID="e4dfae5f2889c6ce6240644308bfe756948a370f74a5fed0b606c1692fd1b9ae"
Aug 01 00:23:16.767084 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:23:16.767036    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4dfae5f2889c6ce6240644308bfe756948a370f74a5fed0b606c1692fd1b9ae\": container with ID starting with e4dfae5f2889c6ce6240644308bfe756948a370f74a5fed0b606c1692fd1b9ae not found: ID does not exist" containerID="e4dfae5f2889c6ce6240644308bfe756948a370f74a5fed0b606c1692fd1b9ae"
Aug 01 00:23:16.767126 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.767081    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4dfae5f2889c6ce6240644308bfe756948a370f74a5fed0b606c1692fd1b9ae"} err="failed to get container status \"e4dfae5f2889c6ce6240644308bfe756948a370f74a5fed0b606c1692fd1b9ae\": rpc error: code = NotFound desc = could not find container \"e4dfae5f2889c6ce6240644308bfe756948a370f74a5fed0b606c1692fd1b9ae\": container with ID starting with e4dfae5f2889c6ce6240644308bfe756948a370f74a5fed0b606c1692fd1b9ae not found: ID does not exist"
Aug 01 00:23:16.767126 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.767100    2546 scope.go:117] "RemoveContainer" containerID="0213dd4705349d29006a74966f0e1172b29282644cd2d419e5a392dcabff188d"
Aug 01 00:23:16.767427 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:23:16.767402    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0213dd4705349d29006a74966f0e1172b29282644cd2d419e5a392dcabff188d\": container with ID starting with 0213dd4705349d29006a74966f0e1172b29282644cd2d419e5a392dcabff188d not found: ID does not exist" containerID="0213dd4705349d29006a74966f0e1172b29282644cd2d419e5a392dcabff188d"
Aug 01 00:23:16.767533 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.767435    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0213dd4705349d29006a74966f0e1172b29282644cd2d419e5a392dcabff188d"} err="failed to get container status \"0213dd4705349d29006a74966f0e1172b29282644cd2d419e5a392dcabff188d\": rpc error: code = NotFound desc = could not find container \"0213dd4705349d29006a74966f0e1172b29282644cd2d419e5a392dcabff188d\": container with ID starting with 0213dd4705349d29006a74966f0e1172b29282644cd2d419e5a392dcabff188d not found: ID does not exist"
Aug 01 00:23:16.767533 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.767459    2546 scope.go:117] "RemoveContainer" containerID="f7851e944d8569554debfcdb7eea71a8e5075b970951d16e6f2e1cf5479fff53"
Aug 01 00:23:16.767729 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:23:16.767711    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7851e944d8569554debfcdb7eea71a8e5075b970951d16e6f2e1cf5479fff53\": container with ID starting with f7851e944d8569554debfcdb7eea71a8e5075b970951d16e6f2e1cf5479fff53 not found: ID does not exist" containerID="f7851e944d8569554debfcdb7eea71a8e5075b970951d16e6f2e1cf5479fff53"
Aug 01 00:23:16.767784 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:16.767737    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7851e944d8569554debfcdb7eea71a8e5075b970951d16e6f2e1cf5479fff53"} err="failed to get container status \"f7851e944d8569554debfcdb7eea71a8e5075b970951d16e6f2e1cf5479fff53\": rpc error: code = NotFound desc = could not find container \"f7851e944d8569554debfcdb7eea71a8e5075b970951d16e6f2e1cf5479fff53\": container with ID starting with f7851e944d8569554debfcdb7eea71a8e5075b970951d16e6f2e1cf5479fff53 not found: ID does not exist"
Aug 01 00:23:18.079415 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:18.079383    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" path="/var/lib/kubelet/pods/8ab31c82-b82d-4675-ac1e-0b69877cc222/volumes"
Aug 01 00:23:58.105459 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:58.105425    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x" event={"ID":"5f882bb7-a819-4f26-ad6f-fd12d7948a84","Type":"ContainerStarted","Data":"48dcce5dc0a11e9b76934b3c9181ee1d7df221d18993b5ddc0f4bf928c66f0fe"}
Aug 01 00:23:58.105951 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:58.105485    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x" event={"ID":"5f882bb7-a819-4f26-ad6f-fd12d7948a84","Type":"ContainerStarted","Data":"065181f610d3c2bdf42c913f0d289ddc9d0d2e7f116bd5e85e23917879f88f3a"}
Aug 01 00:23:58.105951 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:58.105655    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:23:58.105951 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:58.105798    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:23:58.107172 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:58.107144    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x" podUID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Aug 01 00:23:58.126893 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:58.126847    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x" podStartSLOduration=2.909997328 podStartE2EDuration="2m0.126830572s" totalImagesPullingTime="1m53.726906001s" totalInitContainerRuntime="3.489927243s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:21:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:23:58.12434041 +0000 UTC m=+2268.583803223" watchObservedRunningTime="2026-08-01 00:23:58.126830572 +0000 UTC m=+2268.586293385"
Aug 01 00:23:59.109941 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:23:59.109894    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x" podUID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Aug 01 00:24:04.114781 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:04.114751    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:24:04.115557 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:04.115538    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:24:10.305656 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:10.305626    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"]
Aug 01 00:24:10.306137 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:10.305842    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x" podUID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" containerName="kserve-container" containerID="cri-o://065181f610d3c2bdf42c913f0d289ddc9d0d2e7f116bd5e85e23917879f88f3a" gracePeriod=30
Aug 01 00:24:10.306137 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:10.305889    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x" podUID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" containerName="kube-rbac-proxy" containerID="cri-o://48dcce5dc0a11e9b76934b3c9181ee1d7df221d18993b5ddc0f4bf928c66f0fe" gracePeriod=30
Aug 01 00:24:10.329024 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:10.328997    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:11.156857 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:11.156825    2546 generic.go:356] "Generic (PLEG): container finished" podID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" containerID="48dcce5dc0a11e9b76934b3c9181ee1d7df221d18993b5ddc0f4bf928c66f0fe" exitCode=2
Aug 01 00:24:11.157031 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:11.156848    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x" event={"ID":"5f882bb7-a819-4f26-ad6f-fd12d7948a84","Type":"ContainerDied","Data":"48dcce5dc0a11e9b76934b3c9181ee1d7df221d18993b5ddc0f4bf928c66f0fe"}
Aug 01 00:24:12.987358 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:12.987280    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:24:13.086788 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.086754    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjmmg\" (UniqueName: \"kubernetes.io/projected/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kube-api-access-cjmmg\") pod \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") "
Aug 01 00:24:13.086996 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.086795    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kube-rbac-proxy-sar-config\") pod \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") "
Aug 01 00:24:13.086996 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.086846    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f882bb7-a819-4f26-ad6f-fd12d7948a84-proxy-tls\") pod \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") "
Aug 01 00:24:13.086996 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.086862    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kserve-provision-location\") pod \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") "
Aug 01 00:24:13.086996 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.086925    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f882bb7-a819-4f26-ad6f-fd12d7948a84-cabundle-cert\") pod \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\" (UID: \"5f882bb7-a819-4f26-ad6f-fd12d7948a84\") "
Aug 01 00:24:13.087315 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.087277    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5f882bb7-a819-4f26-ad6f-fd12d7948a84" (UID: "5f882bb7-a819-4f26-ad6f-fd12d7948a84"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:24:13.087432 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.087368    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5f882bb7-a819-4f26-ad6f-fd12d7948a84" (UID: "5f882bb7-a819-4f26-ad6f-fd12d7948a84"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:24:13.087432 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.087409    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f882bb7-a819-4f26-ad6f-fd12d7948a84-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5f882bb7-a819-4f26-ad6f-fd12d7948a84" (UID: "5f882bb7-a819-4f26-ad6f-fd12d7948a84"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:24:13.088913 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.088893    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f882bb7-a819-4f26-ad6f-fd12d7948a84-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f882bb7-a819-4f26-ad6f-fd12d7948a84" (UID: "5f882bb7-a819-4f26-ad6f-fd12d7948a84"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:24:13.089000 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.088961    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kube-api-access-cjmmg" (OuterVolumeSpecName: "kube-api-access-cjmmg") pod "5f882bb7-a819-4f26-ad6f-fd12d7948a84" (UID: "5f882bb7-a819-4f26-ad6f-fd12d7948a84"). InnerVolumeSpecName "kube-api-access-cjmmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:24:13.166366 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.166329    2546 generic.go:356] "Generic (PLEG): container finished" podID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" containerID="065181f610d3c2bdf42c913f0d289ddc9d0d2e7f116bd5e85e23917879f88f3a" exitCode=0
Aug 01 00:24:13.166552 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.166383    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"
Aug 01 00:24:13.166552 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.166387    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x" event={"ID":"5f882bb7-a819-4f26-ad6f-fd12d7948a84","Type":"ContainerDied","Data":"065181f610d3c2bdf42c913f0d289ddc9d0d2e7f116bd5e85e23917879f88f3a"}
Aug 01 00:24:13.166552 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.166512    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x" event={"ID":"5f882bb7-a819-4f26-ad6f-fd12d7948a84","Type":"ContainerDied","Data":"452860ee54f48efc9ef07586aab6dc378f6fc90f334f061facebed36e37934fe"}
Aug 01 00:24:13.166552 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.166538    2546 scope.go:117] "RemoveContainer" containerID="48dcce5dc0a11e9b76934b3c9181ee1d7df221d18993b5ddc0f4bf928c66f0fe"
Aug 01 00:24:13.180984 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.180964    2546 scope.go:117] "RemoveContainer" containerID="065181f610d3c2bdf42c913f0d289ddc9d0d2e7f116bd5e85e23917879f88f3a"
Aug 01 00:24:13.187691 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.187664    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f882bb7-a819-4f26-ad6f-fd12d7948a84-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:24:13.187794 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.187696    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjmmg\" (UniqueName: \"kubernetes.io/projected/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kube-api-access-cjmmg\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:24:13.187794 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.187711    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:24:13.187794 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.187724    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f882bb7-a819-4f26-ad6f-fd12d7948a84-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:24:13.187794 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.187736    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f882bb7-a819-4f26-ad6f-fd12d7948a84-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:24:13.188889 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.188868    2546 scope.go:117] "RemoveContainer" containerID="d51b59d51bc8c04b67d485502034164a0b069b4c457f96a7259370428d998dd6"
Aug 01 00:24:13.194371 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.194348    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"]
Aug 01 00:24:13.196787 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.196764    2546 scope.go:117] "RemoveContainer" containerID="48dcce5dc0a11e9b76934b3c9181ee1d7df221d18993b5ddc0f4bf928c66f0fe"
Aug 01 00:24:13.197067 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:24:13.197045    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48dcce5dc0a11e9b76934b3c9181ee1d7df221d18993b5ddc0f4bf928c66f0fe\": container with ID starting with 48dcce5dc0a11e9b76934b3c9181ee1d7df221d18993b5ddc0f4bf928c66f0fe not found: ID does not exist" containerID="48dcce5dc0a11e9b76934b3c9181ee1d7df221d18993b5ddc0f4bf928c66f0fe"
Aug 01 00:24:13.197149 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.197075    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48dcce5dc0a11e9b76934b3c9181ee1d7df221d18993b5ddc0f4bf928c66f0fe"} err="failed to get container status \"48dcce5dc0a11e9b76934b3c9181ee1d7df221d18993b5ddc0f4bf928c66f0fe\": rpc error: code = NotFound desc = could not find container \"48dcce5dc0a11e9b76934b3c9181ee1d7df221d18993b5ddc0f4bf928c66f0fe\": container with ID starting with 48dcce5dc0a11e9b76934b3c9181ee1d7df221d18993b5ddc0f4bf928c66f0fe not found: ID does not exist"
Aug 01 00:24:13.197149 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.197094    2546 scope.go:117] "RemoveContainer" containerID="065181f610d3c2bdf42c913f0d289ddc9d0d2e7f116bd5e85e23917879f88f3a"
Aug 01 00:24:13.197349 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:24:13.197326    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"065181f610d3c2bdf42c913f0d289ddc9d0d2e7f116bd5e85e23917879f88f3a\": container with ID starting with 065181f610d3c2bdf42c913f0d289ddc9d0d2e7f116bd5e85e23917879f88f3a not found: ID does not exist" containerID="065181f610d3c2bdf42c913f0d289ddc9d0d2e7f116bd5e85e23917879f88f3a"
Aug 01 00:24:13.197468 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.197359    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"065181f610d3c2bdf42c913f0d289ddc9d0d2e7f116bd5e85e23917879f88f3a"} err="failed to get container status \"065181f610d3c2bdf42c913f0d289ddc9d0d2e7f116bd5e85e23917879f88f3a\": rpc error: code = NotFound desc = could not find container \"065181f610d3c2bdf42c913f0d289ddc9d0d2e7f116bd5e85e23917879f88f3a\": container with ID starting with 065181f610d3c2bdf42c913f0d289ddc9d0d2e7f116bd5e85e23917879f88f3a not found: ID does not exist"
Aug 01 00:24:13.197468 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.197382    2546 scope.go:117] "RemoveContainer" containerID="d51b59d51bc8c04b67d485502034164a0b069b4c457f96a7259370428d998dd6"
Aug 01 00:24:13.197649 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:24:13.197631    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51b59d51bc8c04b67d485502034164a0b069b4c457f96a7259370428d998dd6\": container with ID starting with d51b59d51bc8c04b67d485502034164a0b069b4c457f96a7259370428d998dd6 not found: ID does not exist" containerID="d51b59d51bc8c04b67d485502034164a0b069b4c457f96a7259370428d998dd6"
Aug 01 00:24:13.197711 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.197652    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51b59d51bc8c04b67d485502034164a0b069b4c457f96a7259370428d998dd6"} err="failed to get container status \"d51b59d51bc8c04b67d485502034164a0b069b4c457f96a7259370428d998dd6\": rpc error: code = NotFound desc = could not find container \"d51b59d51bc8c04b67d485502034164a0b069b4c457f96a7259370428d998dd6\": container with ID starting with d51b59d51bc8c04b67d485502034164a0b069b4c457f96a7259370428d998dd6 not found: ID does not exist"
Aug 01 00:24:13.198586 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:13.198562    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jfc6x"]
Aug 01 00:24:14.075410 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:14.075383    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" path="/var/lib/kubelet/pods/5f882bb7-a819-4f26-ad6f-fd12d7948a84/volumes"
Aug 01 00:24:15.412667 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.412636    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:15.448548 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.448513    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"]
Aug 01 00:24:15.448855 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.448841    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" containerName="kserve-container"
Aug 01 00:24:15.448899 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.448857    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" containerName="kserve-container"
Aug 01 00:24:15.448899 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.448869    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" containerName="storage-initializer"
Aug 01 00:24:15.448899 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.448874    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" containerName="storage-initializer"
Aug 01 00:24:15.448899 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.448882    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kube-rbac-proxy"
Aug 01 00:24:15.448899 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.448887    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kube-rbac-proxy"
Aug 01 00:24:15.448899 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.448900    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="storage-initializer"
Aug 01 00:24:15.449093 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.448905    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="storage-initializer"
Aug 01 00:24:15.449093 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.448912    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container"
Aug 01 00:24:15.449093 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.448917    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container"
Aug 01 00:24:15.449093 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.448924    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" containerName="kube-rbac-proxy"
Aug 01 00:24:15.449093 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.448938    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" containerName="kube-rbac-proxy"
Aug 01 00:24:15.449093 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.448996    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kserve-container"
Aug 01 00:24:15.449093 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.449004    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" containerName="kserve-container"
Aug 01 00:24:15.449093 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.449010    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab31c82-b82d-4675-ac1e-0b69877cc222" containerName="kube-rbac-proxy"
Aug 01 00:24:15.449093 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.449016    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f882bb7-a819-4f26-ad6f-fd12d7948a84" containerName="kube-rbac-proxy"
Aug 01 00:24:15.452000 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.451976    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:15.454874 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.454781    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:24:15.454994 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.454902    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:24:15.455496 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.455482    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:24:15.455556 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.455482    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:24:15.456010 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.455997    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Aug 01 00:24:15.456077 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.456026    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Aug 01 00:24:15.456127 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.456074    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:24:15.465729 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.465702    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"]
Aug 01 00:24:15.504338 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.504295    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c64cf98-b833-4c8d-9382-5c601481c89a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:15.504338 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.504338    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzmj2\" (UniqueName: \"kubernetes.io/projected/8c64cf98-b833-4c8d-9382-5c601481c89a-kube-api-access-bzmj2\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:15.504560 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.504412    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c64cf98-b833-4c8d-9382-5c601481c89a-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:15.504560 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.504466    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c64cf98-b833-4c8d-9382-5c601481c89a-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:15.504560 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.504531    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c64cf98-b833-4c8d-9382-5c601481c89a-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:15.605741 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.605696    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c64cf98-b833-4c8d-9382-5c601481c89a-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:15.605741 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.605745    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c64cf98-b833-4c8d-9382-5c601481c89a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:15.605977 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.605765    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzmj2\" (UniqueName: \"kubernetes.io/projected/8c64cf98-b833-4c8d-9382-5c601481c89a-kube-api-access-bzmj2\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:15.605977 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.605879    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c64cf98-b833-4c8d-9382-5c601481c89a-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:15.605977 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.605934    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c64cf98-b833-4c8d-9382-5c601481c89a-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:15.606148 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:24:15.606060    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-predictor-serving-cert: secret "isvc-xgboost-predictor-serving-cert" not found
Aug 01 00:24:15.606209 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:24:15.606145    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c64cf98-b833-4c8d-9382-5c601481c89a-proxy-tls podName:8c64cf98-b833-4c8d-9382-5c601481c89a nodeName:}" failed. No retries permitted until 2026-08-01 00:24:16.106122736 +0000 UTC m=+2286.565585530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c64cf98-b833-4c8d-9382-5c601481c89a-proxy-tls") pod "isvc-xgboost-predictor-7566d65bf7-8rsnt" (UID: "8c64cf98-b833-4c8d-9382-5c601481c89a") : secret "isvc-xgboost-predictor-serving-cert" not found
Aug 01 00:24:15.606383 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.606360    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c64cf98-b833-4c8d-9382-5c601481c89a-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:15.606452 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.606422    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c64cf98-b833-4c8d-9382-5c601481c89a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:15.606532 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.606517    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c64cf98-b833-4c8d-9382-5c601481c89a-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:15.614204 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.614181    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzmj2\" (UniqueName: \"kubernetes.io/projected/8c64cf98-b833-4c8d-9382-5c601481c89a-kube-api-access-bzmj2\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:15.681665 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:15.681554    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:16.109673 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:16.109589    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c64cf98-b833-4c8d-9382-5c601481c89a-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:16.111991 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:16.111966    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c64cf98-b833-4c8d-9382-5c601481c89a-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-8rsnt\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:16.362021 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:16.361987    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:16.491340 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:16.491313    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"]
Aug 01 00:24:16.493187 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:24:16.493161    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c64cf98_b833_4c8d_9382_5c601481c89a.slice/crio-a7a24ba4efa4c9e2a729e70ff357a5f061af18dd26177ccc74947a9c77790bc2 WatchSource:0}: Error finding container a7a24ba4efa4c9e2a729e70ff357a5f061af18dd26177ccc74947a9c77790bc2: Status 404 returned error can't find the container with id a7a24ba4efa4c9e2a729e70ff357a5f061af18dd26177ccc74947a9c77790bc2
Aug 01 00:24:17.184339 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:17.184302    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" event={"ID":"8c64cf98-b833-4c8d-9382-5c601481c89a","Type":"ContainerStarted","Data":"ad56d007dac600403da1bdd4fd94d297147f77c0cf336c694b5cdf45aee3bec7"}
Aug 01 00:24:17.184555 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:17.184368    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" event={"ID":"8c64cf98-b833-4c8d-9382-5c601481c89a","Type":"ContainerStarted","Data":"a7a24ba4efa4c9e2a729e70ff357a5f061af18dd26177ccc74947a9c77790bc2"}
Aug 01 00:24:17.313197 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:17.313155    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:17.413919 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:17.413863    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:21.200627 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:21.200593    2546 generic.go:356] "Generic (PLEG): container finished" podID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerID="ad56d007dac600403da1bdd4fd94d297147f77c0cf336c694b5cdf45aee3bec7" exitCode=0
Aug 01 00:24:21.200989 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:21.200664    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" event={"ID":"8c64cf98-b833-4c8d-9382-5c601481c89a","Type":"ContainerDied","Data":"ad56d007dac600403da1bdd4fd94d297147f77c0cf336c694b5cdf45aee3bec7"}
Aug 01 00:24:27.653997 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:27.653951    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:27.783936 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:27.783878    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:42.303931 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:42.303892    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" event={"ID":"8c64cf98-b833-4c8d-9382-5c601481c89a","Type":"ContainerStarted","Data":"92730a0006421310ba3ca4ec092cc19aaa03156dd7451bed93e3a8b819a8e7da"}
Aug 01 00:24:42.304325 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:42.303963    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" event={"ID":"8c64cf98-b833-4c8d-9382-5c601481c89a","Type":"ContainerStarted","Data":"73f687de22067f80f6334aeb634301865f44ae31dade22a17e639fe744052d8e"}
Aug 01 00:24:42.304325 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:42.304164    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:42.323937 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:42.323889    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" podStartSLOduration=3.167989778 podStartE2EDuration="27.323874322s" totalImagesPullingTime="20.308789343s" totalInitContainerRuntime="3.847095201s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:24:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:24:42.321303693 +0000 UTC m=+2312.780766496" watchObservedRunningTime="2026-08-01 00:24:42.323874322 +0000 UTC m=+2312.783337134"
Aug 01 00:24:43.308314 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:43.308281    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:43.309821 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:43.309796    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Aug 01 00:24:44.312310 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:44.312239    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Aug 01 00:24:46.134854 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:46.134825    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:24:49.316676 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:49.316649    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:24:49.317277 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:49.317232    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Aug 01 00:24:59.317172 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:24:59.317064    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Aug 01 00:25:09.317573 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:09.317495    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Aug 01 00:25:19.317085 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:19.317042    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Aug 01 00:25:29.318138 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:29.318096    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Aug 01 00:25:39.317917 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:39.317889    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:25:45.562820 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:45.562782    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"]
Aug 01 00:25:45.563224 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:45.563079    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="kserve-container" containerID="cri-o://73f687de22067f80f6334aeb634301865f44ae31dade22a17e639fe744052d8e" gracePeriod=30
Aug 01 00:25:45.563224 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:45.563113    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="kube-rbac-proxy" containerID="cri-o://92730a0006421310ba3ca4ec092cc19aaa03156dd7451bed93e3a8b819a8e7da" gracePeriod=30
Aug 01 00:25:45.577940 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:45.577917    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:46.560191 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:46.560158    2546 generic.go:356] "Generic (PLEG): container finished" podID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerID="92730a0006421310ba3ca4ec092cc19aaa03156dd7451bed93e3a8b819a8e7da" exitCode=2
Aug 01 00:25:46.560384 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:46.560234    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" event={"ID":"8c64cf98-b833-4c8d-9382-5c601481c89a","Type":"ContainerDied","Data":"92730a0006421310ba3ca4ec092cc19aaa03156dd7451bed93e3a8b819a8e7da"}
Aug 01 00:25:48.898750 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:48.898727    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:25:49.035756 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.035675    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c64cf98-b833-4c8d-9382-5c601481c89a-proxy-tls\") pod \"8c64cf98-b833-4c8d-9382-5c601481c89a\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") "
Aug 01 00:25:49.035756 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.035745    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c64cf98-b833-4c8d-9382-5c601481c89a-kserve-provision-location\") pod \"8c64cf98-b833-4c8d-9382-5c601481c89a\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") "
Aug 01 00:25:49.035984 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.035785    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzmj2\" (UniqueName: \"kubernetes.io/projected/8c64cf98-b833-4c8d-9382-5c601481c89a-kube-api-access-bzmj2\") pod \"8c64cf98-b833-4c8d-9382-5c601481c89a\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") "
Aug 01 00:25:49.035984 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.035801    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c64cf98-b833-4c8d-9382-5c601481c89a-cabundle-cert\") pod \"8c64cf98-b833-4c8d-9382-5c601481c89a\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") "
Aug 01 00:25:49.035984 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.035859    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c64cf98-b833-4c8d-9382-5c601481c89a-kube-rbac-proxy-sar-config\") pod \"8c64cf98-b833-4c8d-9382-5c601481c89a\" (UID: \"8c64cf98-b833-4c8d-9382-5c601481c89a\") "
Aug 01 00:25:49.036191 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.036108    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c64cf98-b833-4c8d-9382-5c601481c89a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c64cf98-b833-4c8d-9382-5c601481c89a" (UID: "8c64cf98-b833-4c8d-9382-5c601481c89a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:25:49.036243 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.036217    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c64cf98-b833-4c8d-9382-5c601481c89a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c64cf98-b833-4c8d-9382-5c601481c89a" (UID: "8c64cf98-b833-4c8d-9382-5c601481c89a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:25:49.036301 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.036270    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c64cf98-b833-4c8d-9382-5c601481c89a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c64cf98-b833-4c8d-9382-5c601481c89a" (UID: "8c64cf98-b833-4c8d-9382-5c601481c89a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:25:49.037844 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.037824    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c64cf98-b833-4c8d-9382-5c601481c89a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c64cf98-b833-4c8d-9382-5c601481c89a" (UID: "8c64cf98-b833-4c8d-9382-5c601481c89a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:25:49.037844 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.037830    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c64cf98-b833-4c8d-9382-5c601481c89a-kube-api-access-bzmj2" (OuterVolumeSpecName: "kube-api-access-bzmj2") pod "8c64cf98-b833-4c8d-9382-5c601481c89a" (UID: "8c64cf98-b833-4c8d-9382-5c601481c89a"). InnerVolumeSpecName "kube-api-access-bzmj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:25:49.137455 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.137422    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c64cf98-b833-4c8d-9382-5c601481c89a-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:25:49.137455 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.137450    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c64cf98-b833-4c8d-9382-5c601481c89a-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:25:49.137455 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.137461    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzmj2\" (UniqueName: \"kubernetes.io/projected/8c64cf98-b833-4c8d-9382-5c601481c89a-kube-api-access-bzmj2\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:25:49.137663 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.137473    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c64cf98-b833-4c8d-9382-5c601481c89a-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:25:49.137663 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.137486    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c64cf98-b833-4c8d-9382-5c601481c89a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:25:49.574552 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.574514    2546 generic.go:356] "Generic (PLEG): container finished" podID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerID="73f687de22067f80f6334aeb634301865f44ae31dade22a17e639fe744052d8e" exitCode=0
Aug 01 00:25:49.574716 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.574578    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"
Aug 01 00:25:49.574716 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.574601    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" event={"ID":"8c64cf98-b833-4c8d-9382-5c601481c89a","Type":"ContainerDied","Data":"73f687de22067f80f6334aeb634301865f44ae31dade22a17e639fe744052d8e"}
Aug 01 00:25:49.574716 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.574663    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt" event={"ID":"8c64cf98-b833-4c8d-9382-5c601481c89a","Type":"ContainerDied","Data":"a7a24ba4efa4c9e2a729e70ff357a5f061af18dd26177ccc74947a9c77790bc2"}
Aug 01 00:25:49.574716 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.574692    2546 scope.go:117] "RemoveContainer" containerID="92730a0006421310ba3ca4ec092cc19aaa03156dd7451bed93e3a8b819a8e7da"
Aug 01 00:25:49.582877 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.582860    2546 scope.go:117] "RemoveContainer" containerID="73f687de22067f80f6334aeb634301865f44ae31dade22a17e639fe744052d8e"
Aug 01 00:25:49.591945 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.591353    2546 scope.go:117] "RemoveContainer" containerID="ad56d007dac600403da1bdd4fd94d297147f77c0cf336c694b5cdf45aee3bec7"
Aug 01 00:25:49.595609 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.595588    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"]
Aug 01 00:25:49.599537 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.599521    2546 scope.go:117] "RemoveContainer" containerID="92730a0006421310ba3ca4ec092cc19aaa03156dd7451bed93e3a8b819a8e7da"
Aug 01 00:25:49.599815 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:25:49.599798    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92730a0006421310ba3ca4ec092cc19aaa03156dd7451bed93e3a8b819a8e7da\": container with ID starting with 92730a0006421310ba3ca4ec092cc19aaa03156dd7451bed93e3a8b819a8e7da not found: ID does not exist" containerID="92730a0006421310ba3ca4ec092cc19aaa03156dd7451bed93e3a8b819a8e7da"
Aug 01 00:25:49.599856 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.599823    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92730a0006421310ba3ca4ec092cc19aaa03156dd7451bed93e3a8b819a8e7da"} err="failed to get container status \"92730a0006421310ba3ca4ec092cc19aaa03156dd7451bed93e3a8b819a8e7da\": rpc error: code = NotFound desc = could not find container \"92730a0006421310ba3ca4ec092cc19aaa03156dd7451bed93e3a8b819a8e7da\": container with ID starting with 92730a0006421310ba3ca4ec092cc19aaa03156dd7451bed93e3a8b819a8e7da not found: ID does not exist"
Aug 01 00:25:49.599856 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.599840    2546 scope.go:117] "RemoveContainer" containerID="73f687de22067f80f6334aeb634301865f44ae31dade22a17e639fe744052d8e"
Aug 01 00:25:49.600093 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:25:49.600072    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f687de22067f80f6334aeb634301865f44ae31dade22a17e639fe744052d8e\": container with ID starting with 73f687de22067f80f6334aeb634301865f44ae31dade22a17e639fe744052d8e not found: ID does not exist" containerID="73f687de22067f80f6334aeb634301865f44ae31dade22a17e639fe744052d8e"
Aug 01 00:25:49.600093 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.600087    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-8rsnt"]
Aug 01 00:25:49.600215 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.600096    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f687de22067f80f6334aeb634301865f44ae31dade22a17e639fe744052d8e"} err="failed to get container status \"73f687de22067f80f6334aeb634301865f44ae31dade22a17e639fe744052d8e\": rpc error: code = NotFound desc = could not find container \"73f687de22067f80f6334aeb634301865f44ae31dade22a17e639fe744052d8e\": container with ID starting with 73f687de22067f80f6334aeb634301865f44ae31dade22a17e639fe744052d8e not found: ID does not exist"
Aug 01 00:25:49.600215 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.600111    2546 scope.go:117] "RemoveContainer" containerID="ad56d007dac600403da1bdd4fd94d297147f77c0cf336c694b5cdf45aee3bec7"
Aug 01 00:25:49.600387 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:25:49.600367    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad56d007dac600403da1bdd4fd94d297147f77c0cf336c694b5cdf45aee3bec7\": container with ID starting with ad56d007dac600403da1bdd4fd94d297147f77c0cf336c694b5cdf45aee3bec7 not found: ID does not exist" containerID="ad56d007dac600403da1bdd4fd94d297147f77c0cf336c694b5cdf45aee3bec7"
Aug 01 00:25:49.600430 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:49.600401    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad56d007dac600403da1bdd4fd94d297147f77c0cf336c694b5cdf45aee3bec7"} err="failed to get container status \"ad56d007dac600403da1bdd4fd94d297147f77c0cf336c694b5cdf45aee3bec7\": rpc error: code = NotFound desc = could not find container \"ad56d007dac600403da1bdd4fd94d297147f77c0cf336c694b5cdf45aee3bec7\": container with ID starting with ad56d007dac600403da1bdd4fd94d297147f77c0cf336c694b5cdf45aee3bec7 not found: ID does not exist"
Aug 01 00:25:50.076176 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.076143    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" path="/var/lib/kubelet/pods/8c64cf98-b833-4c8d-9382-5c601481c89a/volumes"
Aug 01 00:25:50.720989 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.720956    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:50.754107 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.754067    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"]
Aug 01 00:25:50.754494 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.754476    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="storage-initializer"
Aug 01 00:25:50.754589 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.754495    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="storage-initializer"
Aug 01 00:25:50.754589 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.754519    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="kserve-container"
Aug 01 00:25:50.754589 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.754528    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="kserve-container"
Aug 01 00:25:50.754589 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.754552    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="kube-rbac-proxy"
Aug 01 00:25:50.754589 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.754562    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="kube-rbac-proxy"
Aug 01 00:25:50.754856 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.754649    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="kserve-container"
Aug 01 00:25:50.754856 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.754663    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c64cf98-b833-4c8d-9382-5c601481c89a" containerName="kube-rbac-proxy"
Aug 01 00:25:50.757808 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.757788    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:50.760327 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.760306    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Aug 01 00:25:50.760572 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.760549    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:25:50.760674 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.760549    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Aug 01 00:25:50.760674 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.760578    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:25:50.760674 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.760578    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:25:50.761426 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.761408    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:25:50.761682 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.761662    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:25:50.768492 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.768461    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"]
Aug 01 00:25:50.851351 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.851322    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28a94639-ccbf-484c-b4fe-731f18f7332a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:50.851516 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.851359    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spq8b\" (UniqueName: \"kubernetes.io/projected/28a94639-ccbf-484c-b4fe-731f18f7332a-kube-api-access-spq8b\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:50.851516 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.851389    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28a94639-ccbf-484c-b4fe-731f18f7332a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:50.851516 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.851498    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28a94639-ccbf-484c-b4fe-731f18f7332a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:50.851686 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.851534    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28a94639-ccbf-484c-b4fe-731f18f7332a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:50.951947 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.951905    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28a94639-ccbf-484c-b4fe-731f18f7332a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:50.951947 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.951950    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spq8b\" (UniqueName: \"kubernetes.io/projected/28a94639-ccbf-484c-b4fe-731f18f7332a-kube-api-access-spq8b\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:50.952120 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.951972    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28a94639-ccbf-484c-b4fe-731f18f7332a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:50.952120 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.952024    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28a94639-ccbf-484c-b4fe-731f18f7332a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:50.952120 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.952050    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28a94639-ccbf-484c-b4fe-731f18f7332a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:50.952308 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:25:50.952141    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Aug 01 00:25:50.952308 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:25:50.952235    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28a94639-ccbf-484c-b4fe-731f18f7332a-proxy-tls podName:28a94639-ccbf-484c-b4fe-731f18f7332a nodeName:}" failed. No retries permitted until 2026-08-01 00:25:51.45221412 +0000 UTC m=+2381.911676912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/28a94639-ccbf-484c-b4fe-731f18f7332a-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d" (UID: "28a94639-ccbf-484c-b4fe-731f18f7332a") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Aug 01 00:25:50.952493 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.952474    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28a94639-ccbf-484c-b4fe-731f18f7332a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:50.952684 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.952668    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28a94639-ccbf-484c-b4fe-731f18f7332a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:50.952722 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.952668    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28a94639-ccbf-484c-b4fe-731f18f7332a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:50.962430 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:50.962412    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spq8b\" (UniqueName: \"kubernetes.io/projected/28a94639-ccbf-484c-b4fe-731f18f7332a-kube-api-access-spq8b\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:51.135340 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:51.135308    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:25:51.457547 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:51.457444    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28a94639-ccbf-484c-b4fe-731f18f7332a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:51.459975 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:51.459944    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28a94639-ccbf-484c-b4fe-731f18f7332a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:51.667912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:51.667868    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:51.793453 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:51.793429    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"]
Aug 01 00:25:51.795971 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:25:51.795933    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a94639_ccbf_484c_b4fe_731f18f7332a.slice/crio-4502cb280de569fb04f2eab48bc83977164a27b13ee188afd8d9ea1b6e72f59d WatchSource:0}: Error finding container 4502cb280de569fb04f2eab48bc83977164a27b13ee188afd8d9ea1b6e72f59d: Status 404 returned error can't find the container with id 4502cb280de569fb04f2eab48bc83977164a27b13ee188afd8d9ea1b6e72f59d
Aug 01 00:25:52.588779 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:52.588742    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d" event={"ID":"28a94639-ccbf-484c-b4fe-731f18f7332a","Type":"ContainerStarted","Data":"9db8a45c3cb527483845684fab32325fb3febb96fab688c9fbf301f5b0a287da"}
Aug 01 00:25:52.589168 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:52.588802    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d" event={"ID":"28a94639-ccbf-484c-b4fe-731f18f7332a","Type":"ContainerStarted","Data":"4502cb280de569fb04f2eab48bc83977164a27b13ee188afd8d9ea1b6e72f59d"}
Aug 01 00:25:55.603538 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:55.603505    2546 generic.go:356] "Generic (PLEG): container finished" podID="28a94639-ccbf-484c-b4fe-731f18f7332a" containerID="9db8a45c3cb527483845684fab32325fb3febb96fab688c9fbf301f5b0a287da" exitCode=0
Aug 01 00:25:55.603870 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:55.603576    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d" event={"ID":"28a94639-ccbf-484c-b4fe-731f18f7332a","Type":"ContainerDied","Data":"9db8a45c3cb527483845684fab32325fb3febb96fab688c9fbf301f5b0a287da"}
Aug 01 00:25:56.609630 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:56.609596    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d" event={"ID":"28a94639-ccbf-484c-b4fe-731f18f7332a","Type":"ContainerStarted","Data":"373a711fa73e34c06e8c942c9f2775219b87c4da2140f7570c63dad139c8df3f"}
Aug 01 00:25:56.610004 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:56.609659    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d" event={"ID":"28a94639-ccbf-484c-b4fe-731f18f7332a","Type":"ContainerStarted","Data":"184829b183e4c1e401c8447c5835f9802dfcbbf7a0911668a0b84170ebea1078"}
Aug 01 00:25:56.610004 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:56.609843    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:25:56.633996 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:56.633952    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d" podStartSLOduration=3.037530528 podStartE2EDuration="6.633937149s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.596406621s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:25:56.631625481 +0000 UTC m=+2387.091088307" watchObservedRunningTime="2026-08-01 00:25:56.633937149 +0000 UTC m=+2387.093399961"
Aug 01 00:25:57.613519 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:25:57.613492    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:26:01.705837 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:01.705801    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:26:03.626481 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:03.626452    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:26:06.821312 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:06.821280    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:26:07.272546 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:07.272511    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:26:33.630676 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:33.630648    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:26:40.859488 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:40.859455    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"]
Aug 01 00:26:40.859903 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:40.859706    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d" podUID="28a94639-ccbf-484c-b4fe-731f18f7332a" containerName="kserve-container" containerID="cri-o://184829b183e4c1e401c8447c5835f9802dfcbbf7a0911668a0b84170ebea1078" gracePeriod=30
Aug 01 00:26:40.859903 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:40.859722    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d" podUID="28a94639-ccbf-484c-b4fe-731f18f7332a" containerName="kube-rbac-proxy" containerID="cri-o://373a711fa73e34c06e8c942c9f2775219b87c4da2140f7570c63dad139c8df3f" gracePeriod=30
Aug 01 00:26:40.877893 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:40.877860    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:26:41.799330 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:41.799295    2546 generic.go:356] "Generic (PLEG): container finished" podID="28a94639-ccbf-484c-b4fe-731f18f7332a" containerID="373a711fa73e34c06e8c942c9f2775219b87c4da2140f7570c63dad139c8df3f" exitCode=2
Aug 01 00:26:41.799330 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:41.799337    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d" event={"ID":"28a94639-ccbf-484c-b4fe-731f18f7332a","Type":"ContainerDied","Data":"373a711fa73e34c06e8c942c9f2775219b87c4da2140f7570c63dad139c8df3f"}
Aug 01 00:26:43.620747 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:43.620707    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d" podUID="28a94639-ccbf-484c-b4fe-731f18f7332a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Aug 01 00:26:43.626971 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:43.626936    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d" podUID="28a94639-ccbf-484c-b4fe-731f18f7332a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.54:8080: connect: connection refused"
Aug 01 00:26:45.979662 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:45.979627    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:26:46.030347 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.030314    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"]
Aug 01 00:26:46.033922 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.033905    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.036381 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.036360    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Aug 01 00:26:46.036506 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.036363    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Aug 01 00:26:46.043117 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.043096    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"]
Aug 01 00:26:46.100909 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.100885    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87484a9f-8b31-45b3-b641-79043251ee9d-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.101012 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.100923    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4h7c\" (UniqueName: \"kubernetes.io/projected/87484a9f-8b31-45b3-b641-79043251ee9d-kube-api-access-l4h7c\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.101012 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.100952    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87484a9f-8b31-45b3-b641-79043251ee9d-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.101012 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.101002    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87484a9f-8b31-45b3-b641-79043251ee9d-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.101166 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.101026    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87484a9f-8b31-45b3-b641-79043251ee9d-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.202162 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.202135    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87484a9f-8b31-45b3-b641-79043251ee9d-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.202308 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.202177    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87484a9f-8b31-45b3-b641-79043251ee9d-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.202308 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.202216    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87484a9f-8b31-45b3-b641-79043251ee9d-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.202308 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:26:46.202301    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Aug 01 00:26:46.202458 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.202317    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87484a9f-8b31-45b3-b641-79043251ee9d-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.202458 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:26:46.202365    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87484a9f-8b31-45b3-b641-79043251ee9d-proxy-tls podName:87484a9f-8b31-45b3-b641-79043251ee9d nodeName:}" failed. No retries permitted until 2026-08-01 00:26:46.702350549 +0000 UTC m=+2437.161813339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/87484a9f-8b31-45b3-b641-79043251ee9d-proxy-tls") pod "xgboost-v2-mlserver-predictor-59944bcf4c-7qk68" (UID: "87484a9f-8b31-45b3-b641-79043251ee9d") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Aug 01 00:26:46.202458 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.202384    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4h7c\" (UniqueName: \"kubernetes.io/projected/87484a9f-8b31-45b3-b641-79043251ee9d-kube-api-access-l4h7c\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.202718 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.202697    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87484a9f-8b31-45b3-b641-79043251ee9d-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.203006 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.202984    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87484a9f-8b31-45b3-b641-79043251ee9d-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.203044 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.202984    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87484a9f-8b31-45b3-b641-79043251ee9d-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.211340 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.211324    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4h7c\" (UniqueName: \"kubernetes.io/projected/87484a9f-8b31-45b3-b641-79043251ee9d-kube-api-access-l4h7c\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.352533 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.352469    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:26:46.699370 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.699348    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:26:46.705770 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.705754    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87484a9f-8b31-45b3-b641-79043251ee9d-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.707883 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.707867    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87484a9f-8b31-45b3-b641-79043251ee9d-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7qk68\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:46.807052 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.807027    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spq8b\" (UniqueName: \"kubernetes.io/projected/28a94639-ccbf-484c-b4fe-731f18f7332a-kube-api-access-spq8b\") pod \"28a94639-ccbf-484c-b4fe-731f18f7332a\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") "
Aug 01 00:26:46.807184 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.807057    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28a94639-ccbf-484c-b4fe-731f18f7332a-kserve-provision-location\") pod \"28a94639-ccbf-484c-b4fe-731f18f7332a\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") "
Aug 01 00:26:46.807184 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.807078    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28a94639-ccbf-484c-b4fe-731f18f7332a-kube-rbac-proxy-sar-config\") pod \"28a94639-ccbf-484c-b4fe-731f18f7332a\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") "
Aug 01 00:26:46.807184 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.807099    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28a94639-ccbf-484c-b4fe-731f18f7332a-cabundle-cert\") pod \"28a94639-ccbf-484c-b4fe-731f18f7332a\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") "
Aug 01 00:26:46.807184 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.807165    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28a94639-ccbf-484c-b4fe-731f18f7332a-proxy-tls\") pod \"28a94639-ccbf-484c-b4fe-731f18f7332a\" (UID: \"28a94639-ccbf-484c-b4fe-731f18f7332a\") "
Aug 01 00:26:46.807438 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.807411    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a94639-ccbf-484c-b4fe-731f18f7332a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "28a94639-ccbf-484c-b4fe-731f18f7332a" (UID: "28a94639-ccbf-484c-b4fe-731f18f7332a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:26:46.807489 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.807439    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a94639-ccbf-484c-b4fe-731f18f7332a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "28a94639-ccbf-484c-b4fe-731f18f7332a" (UID: "28a94639-ccbf-484c-b4fe-731f18f7332a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:26:46.807522 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.807504    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a94639-ccbf-484c-b4fe-731f18f7332a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "28a94639-ccbf-484c-b4fe-731f18f7332a" (UID: "28a94639-ccbf-484c-b4fe-731f18f7332a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:26:46.809039 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.809018    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a94639-ccbf-484c-b4fe-731f18f7332a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "28a94639-ccbf-484c-b4fe-731f18f7332a" (UID: "28a94639-ccbf-484c-b4fe-731f18f7332a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:26:46.809113 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.809073    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a94639-ccbf-484c-b4fe-731f18f7332a-kube-api-access-spq8b" (OuterVolumeSpecName: "kube-api-access-spq8b") pod "28a94639-ccbf-484c-b4fe-731f18f7332a" (UID: "28a94639-ccbf-484c-b4fe-731f18f7332a"). InnerVolumeSpecName "kube-api-access-spq8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:26:46.826464 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.826438    2546 generic.go:356] "Generic (PLEG): container finished" podID="28a94639-ccbf-484c-b4fe-731f18f7332a" containerID="184829b183e4c1e401c8447c5835f9802dfcbbf7a0911668a0b84170ebea1078" exitCode=0
Aug 01 00:26:46.826538 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.826498    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"
Aug 01 00:26:46.826538 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.826506    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d" event={"ID":"28a94639-ccbf-484c-b4fe-731f18f7332a","Type":"ContainerDied","Data":"184829b183e4c1e401c8447c5835f9802dfcbbf7a0911668a0b84170ebea1078"}
Aug 01 00:26:46.826615 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.826563    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d" event={"ID":"28a94639-ccbf-484c-b4fe-731f18f7332a","Type":"ContainerDied","Data":"4502cb280de569fb04f2eab48bc83977164a27b13ee188afd8d9ea1b6e72f59d"}
Aug 01 00:26:46.826615 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.826600    2546 scope.go:117] "RemoveContainer" containerID="373a711fa73e34c06e8c942c9f2775219b87c4da2140f7570c63dad139c8df3f"
Aug 01 00:26:46.836993 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.836975    2546 scope.go:117] "RemoveContainer" containerID="184829b183e4c1e401c8447c5835f9802dfcbbf7a0911668a0b84170ebea1078"
Aug 01 00:26:46.844056 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.844040    2546 scope.go:117] "RemoveContainer" containerID="9db8a45c3cb527483845684fab32325fb3febb96fab688c9fbf301f5b0a287da"
Aug 01 00:26:46.848634 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.848616    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"]
Aug 01 00:26:46.851936 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.851921    2546 scope.go:117] "RemoveContainer" containerID="373a711fa73e34c06e8c942c9f2775219b87c4da2140f7570c63dad139c8df3f"
Aug 01 00:26:46.852181 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:26:46.852163    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373a711fa73e34c06e8c942c9f2775219b87c4da2140f7570c63dad139c8df3f\": container with ID starting with 373a711fa73e34c06e8c942c9f2775219b87c4da2140f7570c63dad139c8df3f not found: ID does not exist" containerID="373a711fa73e34c06e8c942c9f2775219b87c4da2140f7570c63dad139c8df3f"
Aug 01 00:26:46.852240 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.852189    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373a711fa73e34c06e8c942c9f2775219b87c4da2140f7570c63dad139c8df3f"} err="failed to get container status \"373a711fa73e34c06e8c942c9f2775219b87c4da2140f7570c63dad139c8df3f\": rpc error: code = NotFound desc = could not find container \"373a711fa73e34c06e8c942c9f2775219b87c4da2140f7570c63dad139c8df3f\": container with ID starting with 373a711fa73e34c06e8c942c9f2775219b87c4da2140f7570c63dad139c8df3f not found: ID does not exist"
Aug 01 00:26:46.852240 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.852205    2546 scope.go:117] "RemoveContainer" containerID="184829b183e4c1e401c8447c5835f9802dfcbbf7a0911668a0b84170ebea1078"
Aug 01 00:26:46.852424 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:26:46.852409    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184829b183e4c1e401c8447c5835f9802dfcbbf7a0911668a0b84170ebea1078\": container with ID starting with 184829b183e4c1e401c8447c5835f9802dfcbbf7a0911668a0b84170ebea1078 not found: ID does not exist" containerID="184829b183e4c1e401c8447c5835f9802dfcbbf7a0911668a0b84170ebea1078"
Aug 01 00:26:46.852479 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.852427    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184829b183e4c1e401c8447c5835f9802dfcbbf7a0911668a0b84170ebea1078"} err="failed to get container status \"184829b183e4c1e401c8447c5835f9802dfcbbf7a0911668a0b84170ebea1078\": rpc error: code = NotFound desc = could not find container \"184829b183e4c1e401c8447c5835f9802dfcbbf7a0911668a0b84170ebea1078\": container with ID starting with 184829b183e4c1e401c8447c5835f9802dfcbbf7a0911668a0b84170ebea1078 not found: ID does not exist"
Aug 01 00:26:46.852479 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.852440    2546 scope.go:117] "RemoveContainer" containerID="9db8a45c3cb527483845684fab32325fb3febb96fab688c9fbf301f5b0a287da"
Aug 01 00:26:46.852479 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.852446    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-qs26d"]
Aug 01 00:26:46.852688 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:26:46.852672    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db8a45c3cb527483845684fab32325fb3febb96fab688c9fbf301f5b0a287da\": container with ID starting with 9db8a45c3cb527483845684fab32325fb3febb96fab688c9fbf301f5b0a287da not found: ID does not exist" containerID="9db8a45c3cb527483845684fab32325fb3febb96fab688c9fbf301f5b0a287da"
Aug 01 00:26:46.852729 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.852694    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db8a45c3cb527483845684fab32325fb3febb96fab688c9fbf301f5b0a287da"} err="failed to get container status \"9db8a45c3cb527483845684fab32325fb3febb96fab688c9fbf301f5b0a287da\": rpc error: code = NotFound desc = could not find container \"9db8a45c3cb527483845684fab32325fb3febb96fab688c9fbf301f5b0a287da\": container with ID starting with 9db8a45c3cb527483845684fab32325fb3febb96fab688c9fbf301f5b0a287da not found: ID does not exist"
Aug 01 00:26:46.908027 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.907981    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spq8b\" (UniqueName: \"kubernetes.io/projected/28a94639-ccbf-484c-b4fe-731f18f7332a-kube-api-access-spq8b\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:26:46.908027 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.908001    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28a94639-ccbf-484c-b4fe-731f18f7332a-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:26:46.908027 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.908011    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28a94639-ccbf-484c-b4fe-731f18f7332a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:26:46.908027 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.908020    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28a94639-ccbf-484c-b4fe-731f18f7332a-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:26:46.908027 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.908029    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28a94639-ccbf-484c-b4fe-731f18f7332a-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:26:46.944580 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:46.944560    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:47.069964 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:47.069939    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"]
Aug 01 00:26:47.071747 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:26:47.071721    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87484a9f_8b31_45b3_b641_79043251ee9d.slice/crio-2d6d89405dea63c26413b24480e700a4c118f10ae4d518401d77a5a903b45f01 WatchSource:0}: Error finding container 2d6d89405dea63c26413b24480e700a4c118f10ae4d518401d77a5a903b45f01: Status 404 returned error can't find the container with id 2d6d89405dea63c26413b24480e700a4c118f10ae4d518401d77a5a903b45f01
Aug 01 00:26:47.831535 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:47.831506    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68" event={"ID":"87484a9f-8b31-45b3-b641-79043251ee9d","Type":"ContainerStarted","Data":"ba4f2e1abe712c41c10a5050e102e6845936863fe87c4ea2151cb84568c98b9d"}
Aug 01 00:26:47.831687 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:47.831559    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68" event={"ID":"87484a9f-8b31-45b3-b641-79043251ee9d","Type":"ContainerStarted","Data":"2d6d89405dea63c26413b24480e700a4c118f10ae4d518401d77a5a903b45f01"}
Aug 01 00:26:48.074920 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:48.074888    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28a94639-ccbf-484c-b4fe-731f18f7332a" path="/var/lib/kubelet/pods/28a94639-ccbf-484c-b4fe-731f18f7332a/volumes"
Aug 01 00:26:51.849582 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:51.849550    2546 generic.go:356] "Generic (PLEG): container finished" podID="87484a9f-8b31-45b3-b641-79043251ee9d" containerID="ba4f2e1abe712c41c10a5050e102e6845936863fe87c4ea2151cb84568c98b9d" exitCode=0
Aug 01 00:26:51.849958 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:51.849623    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68" event={"ID":"87484a9f-8b31-45b3-b641-79043251ee9d","Type":"ContainerDied","Data":"ba4f2e1abe712c41c10a5050e102e6845936863fe87c4ea2151cb84568c98b9d"}
Aug 01 00:26:52.855156 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:52.855123    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68" event={"ID":"87484a9f-8b31-45b3-b641-79043251ee9d","Type":"ContainerStarted","Data":"ab3d4c2f5f978ab3db23d454fd750b4176c9b530228cbcc8fa63750993b79a41"}
Aug 01 00:26:52.855565 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:52.855182    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68" event={"ID":"87484a9f-8b31-45b3-b641-79043251ee9d","Type":"ContainerStarted","Data":"bcaf332a1ee90fa193c7b72cf0360ab6d8e68736c992bc146699a6d38875ea41"}
Aug 01 00:26:52.855565 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:52.855413    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:52.855690 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:52.855564    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:26:52.875075 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:52.875029    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68" podStartSLOduration=3.162452477 podStartE2EDuration="6.875016699s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.712564222s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:26:52.874372526 +0000 UTC m=+2443.333835339" watchObservedRunningTime="2026-08-01 00:26:52.875016699 +0000 UTC m=+2443.334479512"
Aug 01 00:26:58.865661 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:26:58.865630    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:27:28.869982 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:28.869946    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:27:36.089524 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:36.089491    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"]
Aug 01 00:27:36.090079 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:36.089774    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68" podUID="87484a9f-8b31-45b3-b641-79043251ee9d" containerName="kserve-container" containerID="cri-o://bcaf332a1ee90fa193c7b72cf0360ab6d8e68736c992bc146699a6d38875ea41" gracePeriod=30
Aug 01 00:27:36.090079 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:36.089848    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68" podUID="87484a9f-8b31-45b3-b641-79043251ee9d" containerName="kube-rbac-proxy" containerID="cri-o://ab3d4c2f5f978ab3db23d454fd750b4176c9b530228cbcc8fa63750993b79a41" gracePeriod=30
Aug 01 00:27:36.104834 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:36.104806    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:37.030985 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:37.030956    2546 generic.go:356] "Generic (PLEG): container finished" podID="87484a9f-8b31-45b3-b641-79043251ee9d" containerID="ab3d4c2f5f978ab3db23d454fd750b4176c9b530228cbcc8fa63750993b79a41" exitCode=2
Aug 01 00:27:37.031163 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:37.031036    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68" event={"ID":"87484a9f-8b31-45b3-b641-79043251ee9d","Type":"ContainerDied","Data":"ab3d4c2f5f978ab3db23d454fd750b4176c9b530228cbcc8fa63750993b79a41"}
Aug 01 00:27:37.477610 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:37.477579    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:38.861112 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:38.861069    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68" podUID="87484a9f-8b31-45b3-b641-79043251ee9d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Aug 01 00:27:38.866556 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:38.866528    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68" podUID="87484a9f-8b31-45b3-b641-79043251ee9d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.55:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.133.0.55:8080: connect: connection refused"
Aug 01 00:27:41.239552 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.239500    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:41.274095 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.274062    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"]
Aug 01 00:27:41.274407 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.274394    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28a94639-ccbf-484c-b4fe-731f18f7332a" containerName="kserve-container"
Aug 01 00:27:41.274457 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.274408    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="28a94639-ccbf-484c-b4fe-731f18f7332a" containerName="kserve-container"
Aug 01 00:27:41.274457 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.274423    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28a94639-ccbf-484c-b4fe-731f18f7332a" containerName="storage-initializer"
Aug 01 00:27:41.274457 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.274430    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="28a94639-ccbf-484c-b4fe-731f18f7332a" containerName="storage-initializer"
Aug 01 00:27:41.274457 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.274450    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28a94639-ccbf-484c-b4fe-731f18f7332a" containerName="kube-rbac-proxy"
Aug 01 00:27:41.274457 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.274456    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="28a94639-ccbf-484c-b4fe-731f18f7332a" containerName="kube-rbac-proxy"
Aug 01 00:27:41.274614 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.274509    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="28a94639-ccbf-484c-b4fe-731f18f7332a" containerName="kserve-container"
Aug 01 00:27:41.274614 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.274517    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="28a94639-ccbf-484c-b4fe-731f18f7332a" containerName="kube-rbac-proxy"
Aug 01 00:27:41.277728 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.277710    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.280055 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.280035    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Aug 01 00:27:41.280185 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.280052    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Aug 01 00:27:41.287979 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.287956    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"]
Aug 01 00:27:41.362156 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.362122    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk9br\" (UniqueName: \"kubernetes.io/projected/85b07345-6b7d-478f-9e2c-6053ec761612-kube-api-access-hk9br\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.362156 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.362166    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b07345-6b7d-478f-9e2c-6053ec761612-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.362476 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.362206    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85b07345-6b7d-478f-9e2c-6053ec761612-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.362476 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.362231    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85b07345-6b7d-478f-9e2c-6053ec761612-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.362476 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.362248    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85b07345-6b7d-478f-9e2c-6053ec761612-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.462991 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.462961    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9br\" (UniqueName: \"kubernetes.io/projected/85b07345-6b7d-478f-9e2c-6053ec761612-kube-api-access-hk9br\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.463149 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.463007    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b07345-6b7d-478f-9e2c-6053ec761612-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.463149 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:27:41.463133    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Aug 01 00:27:41.463228 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.463140    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85b07345-6b7d-478f-9e2c-6053ec761612-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.463228 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:27:41.463191    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b07345-6b7d-478f-9e2c-6053ec761612-proxy-tls podName:85b07345-6b7d-478f-9e2c-6053ec761612 nodeName:}" failed. No retries permitted until 2026-08-01 00:27:41.963172191 +0000 UTC m=+2492.422634984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/85b07345-6b7d-478f-9e2c-6053ec761612-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-klxcp" (UID: "85b07345-6b7d-478f-9e2c-6053ec761612") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Aug 01 00:27:41.463228 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.463222    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85b07345-6b7d-478f-9e2c-6053ec761612-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.463373 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.463265    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85b07345-6b7d-478f-9e2c-6053ec761612-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.463642 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.463625    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85b07345-6b7d-478f-9e2c-6053ec761612-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.463830 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.463813    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85b07345-6b7d-478f-9e2c-6053ec761612-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.463867 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.463814    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85b07345-6b7d-478f-9e2c-6053ec761612-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.471350 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.471327    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk9br\" (UniqueName: \"kubernetes.io/projected/85b07345-6b7d-478f-9e2c-6053ec761612-kube-api-access-hk9br\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.968668 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:41.968623    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b07345-6b7d-478f-9e2c-6053ec761612-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:41.968854 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:27:41.968774    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Aug 01 00:27:41.968854 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:27:41.968833    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b07345-6b7d-478f-9e2c-6053ec761612-proxy-tls podName:85b07345-6b7d-478f-9e2c-6053ec761612 nodeName:}" failed. No retries permitted until 2026-08-01 00:27:42.968818557 +0000 UTC m=+2493.428281349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/85b07345-6b7d-478f-9e2c-6053ec761612-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-klxcp" (UID: "85b07345-6b7d-478f-9e2c-6053ec761612") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Aug 01 00:27:42.067241 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.067197    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:42.325601 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.325578    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:27:42.473153 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.473116    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87484a9f-8b31-45b3-b641-79043251ee9d-kube-rbac-proxy-sar-config\") pod \"87484a9f-8b31-45b3-b641-79043251ee9d\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") "
Aug 01 00:27:42.473153 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.473155    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87484a9f-8b31-45b3-b641-79043251ee9d-cabundle-cert\") pod \"87484a9f-8b31-45b3-b641-79043251ee9d\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") "
Aug 01 00:27:42.473454 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.473175    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4h7c\" (UniqueName: \"kubernetes.io/projected/87484a9f-8b31-45b3-b641-79043251ee9d-kube-api-access-l4h7c\") pod \"87484a9f-8b31-45b3-b641-79043251ee9d\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") "
Aug 01 00:27:42.473454 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.473203    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87484a9f-8b31-45b3-b641-79043251ee9d-proxy-tls\") pod \"87484a9f-8b31-45b3-b641-79043251ee9d\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") "
Aug 01 00:27:42.473454 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.473263    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87484a9f-8b31-45b3-b641-79043251ee9d-kserve-provision-location\") pod \"87484a9f-8b31-45b3-b641-79043251ee9d\" (UID: \"87484a9f-8b31-45b3-b641-79043251ee9d\") "
Aug 01 00:27:42.473713 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.473687    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87484a9f-8b31-45b3-b641-79043251ee9d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "87484a9f-8b31-45b3-b641-79043251ee9d" (UID: "87484a9f-8b31-45b3-b641-79043251ee9d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:27:42.473764 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.473705    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87484a9f-8b31-45b3-b641-79043251ee9d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87484a9f-8b31-45b3-b641-79043251ee9d" (UID: "87484a9f-8b31-45b3-b641-79043251ee9d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:27:42.473822 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.473761    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87484a9f-8b31-45b3-b641-79043251ee9d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "87484a9f-8b31-45b3-b641-79043251ee9d" (UID: "87484a9f-8b31-45b3-b641-79043251ee9d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:27:42.475384 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.475366    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87484a9f-8b31-45b3-b641-79043251ee9d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "87484a9f-8b31-45b3-b641-79043251ee9d" (UID: "87484a9f-8b31-45b3-b641-79043251ee9d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:27:42.475544 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.475515    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87484a9f-8b31-45b3-b641-79043251ee9d-kube-api-access-l4h7c" (OuterVolumeSpecName: "kube-api-access-l4h7c") pod "87484a9f-8b31-45b3-b641-79043251ee9d" (UID: "87484a9f-8b31-45b3-b641-79043251ee9d"). InnerVolumeSpecName "kube-api-access-l4h7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:27:42.574670 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.574583    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87484a9f-8b31-45b3-b641-79043251ee9d-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:27:42.574670 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.574610    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4h7c\" (UniqueName: \"kubernetes.io/projected/87484a9f-8b31-45b3-b641-79043251ee9d-kube-api-access-l4h7c\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:27:42.574670 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.574620    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87484a9f-8b31-45b3-b641-79043251ee9d-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:27:42.574670 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.574629    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87484a9f-8b31-45b3-b641-79043251ee9d-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:27:42.574670 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.574638    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87484a9f-8b31-45b3-b641-79043251ee9d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:27:42.583356 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.583329    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:42.612571 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.612532    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"]
Aug 01 00:27:42.612881 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.612867    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87484a9f-8b31-45b3-b641-79043251ee9d" containerName="kserve-container"
Aug 01 00:27:42.612947 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.612882    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="87484a9f-8b31-45b3-b641-79043251ee9d" containerName="kserve-container"
Aug 01 00:27:42.612947 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.612891    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87484a9f-8b31-45b3-b641-79043251ee9d" containerName="storage-initializer"
Aug 01 00:27:42.612947 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.612897    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="87484a9f-8b31-45b3-b641-79043251ee9d" containerName="storage-initializer"
Aug 01 00:27:42.612947 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.612909    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87484a9f-8b31-45b3-b641-79043251ee9d" containerName="kube-rbac-proxy"
Aug 01 00:27:42.612947 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.612915    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="87484a9f-8b31-45b3-b641-79043251ee9d" containerName="kube-rbac-proxy"
Aug 01 00:27:42.613108 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.612983    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="87484a9f-8b31-45b3-b641-79043251ee9d" containerName="kube-rbac-proxy"
Aug 01 00:27:42.613108 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.612992    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="87484a9f-8b31-45b3-b641-79043251ee9d" containerName="kserve-container"
Aug 01 00:27:42.616223 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.616203    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:42.618895 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.618873    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Aug 01 00:27:42.619066 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.618873    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Aug 01 00:27:42.627865 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.627839    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"]
Aug 01 00:27:42.675702 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.675662    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c88335-09a1-4426-abaa-ffa022d45366-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:42.675859 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.675714    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62c88335-09a1-4426-abaa-ffa022d45366-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:42.675859 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.675743    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62c88335-09a1-4426-abaa-ffa022d45366-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:42.675859 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.675765    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2sxh\" (UniqueName: \"kubernetes.io/projected/62c88335-09a1-4426-abaa-ffa022d45366-kube-api-access-n2sxh\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:42.675859 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.675795    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62c88335-09a1-4426-abaa-ffa022d45366-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:42.777341 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.777311    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c88335-09a1-4426-abaa-ffa022d45366-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:42.777500 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.777360    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62c88335-09a1-4426-abaa-ffa022d45366-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:42.777500 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.777407    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62c88335-09a1-4426-abaa-ffa022d45366-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:42.777500 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.777434    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2sxh\" (UniqueName: \"kubernetes.io/projected/62c88335-09a1-4426-abaa-ffa022d45366-kube-api-access-n2sxh\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:42.777500 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.777477    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62c88335-09a1-4426-abaa-ffa022d45366-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:42.777709 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:27:42.777531    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Aug 01 00:27:42.777709 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:27:42.777620    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c88335-09a1-4426-abaa-ffa022d45366-proxy-tls podName:62c88335-09a1-4426-abaa-ffa022d45366 nodeName:}" failed. No retries permitted until 2026-08-01 00:27:43.27759893 +0000 UTC m=+2493.737061722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/62c88335-09a1-4426-abaa-ffa022d45366-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv" (UID: "62c88335-09a1-4426-abaa-ffa022d45366") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Aug 01 00:27:42.777853 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.777830    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62c88335-09a1-4426-abaa-ffa022d45366-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:42.778124 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.778106    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62c88335-09a1-4426-abaa-ffa022d45366-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:42.778230 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.778214    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62c88335-09a1-4426-abaa-ffa022d45366-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:42.785633 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.785611    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2sxh\" (UniqueName: \"kubernetes.io/projected/62c88335-09a1-4426-abaa-ffa022d45366-kube-api-access-n2sxh\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:42.907202 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.907163    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:27:42.979388 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.979353    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b07345-6b7d-478f-9e2c-6053ec761612-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:42.981886 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:42.981851    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b07345-6b7d-478f-9e2c-6053ec761612-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-klxcp\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:43.059461 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.059430    2546 generic.go:356] "Generic (PLEG): container finished" podID="87484a9f-8b31-45b3-b641-79043251ee9d" containerID="bcaf332a1ee90fa193c7b72cf0360ab6d8e68736c992bc146699a6d38875ea41" exitCode=0
Aug 01 00:27:43.059635 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.059479    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"
Aug 01 00:27:43.059635 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.059541    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68" event={"ID":"87484a9f-8b31-45b3-b641-79043251ee9d","Type":"ContainerDied","Data":"bcaf332a1ee90fa193c7b72cf0360ab6d8e68736c992bc146699a6d38875ea41"}
Aug 01 00:27:43.059635 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.059588    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68" event={"ID":"87484a9f-8b31-45b3-b641-79043251ee9d","Type":"ContainerDied","Data":"2d6d89405dea63c26413b24480e700a4c118f10ae4d518401d77a5a903b45f01"}
Aug 01 00:27:43.059635 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.059615    2546 scope.go:117] "RemoveContainer" containerID="ab3d4c2f5f978ab3db23d454fd750b4176c9b530228cbcc8fa63750993b79a41"
Aug 01 00:27:43.070281 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.070238    2546 scope.go:117] "RemoveContainer" containerID="bcaf332a1ee90fa193c7b72cf0360ab6d8e68736c992bc146699a6d38875ea41"
Aug 01 00:27:43.078815 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.078795    2546 scope.go:117] "RemoveContainer" containerID="ba4f2e1abe712c41c10a5050e102e6845936863fe87c4ea2151cb84568c98b9d"
Aug 01 00:27:43.086367 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.086335    2546 scope.go:117] "RemoveContainer" containerID="ab3d4c2f5f978ab3db23d454fd750b4176c9b530228cbcc8fa63750993b79a41"
Aug 01 00:27:43.086676 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:27:43.086657    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3d4c2f5f978ab3db23d454fd750b4176c9b530228cbcc8fa63750993b79a41\": container with ID starting with ab3d4c2f5f978ab3db23d454fd750b4176c9b530228cbcc8fa63750993b79a41 not found: ID does not exist" containerID="ab3d4c2f5f978ab3db23d454fd750b4176c9b530228cbcc8fa63750993b79a41"
Aug 01 00:27:43.086755 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.086686    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3d4c2f5f978ab3db23d454fd750b4176c9b530228cbcc8fa63750993b79a41"} err="failed to get container status \"ab3d4c2f5f978ab3db23d454fd750b4176c9b530228cbcc8fa63750993b79a41\": rpc error: code = NotFound desc = could not find container \"ab3d4c2f5f978ab3db23d454fd750b4176c9b530228cbcc8fa63750993b79a41\": container with ID starting with ab3d4c2f5f978ab3db23d454fd750b4176c9b530228cbcc8fa63750993b79a41 not found: ID does not exist"
Aug 01 00:27:43.086755 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.086708    2546 scope.go:117] "RemoveContainer" containerID="bcaf332a1ee90fa193c7b72cf0360ab6d8e68736c992bc146699a6d38875ea41"
Aug 01 00:27:43.086953 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:27:43.086936    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcaf332a1ee90fa193c7b72cf0360ab6d8e68736c992bc146699a6d38875ea41\": container with ID starting with bcaf332a1ee90fa193c7b72cf0360ab6d8e68736c992bc146699a6d38875ea41 not found: ID does not exist" containerID="bcaf332a1ee90fa193c7b72cf0360ab6d8e68736c992bc146699a6d38875ea41"
Aug 01 00:27:43.087002 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.086960    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcaf332a1ee90fa193c7b72cf0360ab6d8e68736c992bc146699a6d38875ea41"} err="failed to get container status \"bcaf332a1ee90fa193c7b72cf0360ab6d8e68736c992bc146699a6d38875ea41\": rpc error: code = NotFound desc = could not find container \"bcaf332a1ee90fa193c7b72cf0360ab6d8e68736c992bc146699a6d38875ea41\": container with ID starting with bcaf332a1ee90fa193c7b72cf0360ab6d8e68736c992bc146699a6d38875ea41 not found: ID does not exist"
Aug 01 00:27:43.087002 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.086976    2546 scope.go:117] "RemoveContainer" containerID="ba4f2e1abe712c41c10a5050e102e6845936863fe87c4ea2151cb84568c98b9d"
Aug 01 00:27:43.087218 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:27:43.087199    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba4f2e1abe712c41c10a5050e102e6845936863fe87c4ea2151cb84568c98b9d\": container with ID starting with ba4f2e1abe712c41c10a5050e102e6845936863fe87c4ea2151cb84568c98b9d not found: ID does not exist" containerID="ba4f2e1abe712c41c10a5050e102e6845936863fe87c4ea2151cb84568c98b9d"
Aug 01 00:27:43.087343 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.087228    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba4f2e1abe712c41c10a5050e102e6845936863fe87c4ea2151cb84568c98b9d"} err="failed to get container status \"ba4f2e1abe712c41c10a5050e102e6845936863fe87c4ea2151cb84568c98b9d\": rpc error: code = NotFound desc = could not find container \"ba4f2e1abe712c41c10a5050e102e6845936863fe87c4ea2151cb84568c98b9d\": container with ID starting with ba4f2e1abe712c41c10a5050e102e6845936863fe87c4ea2151cb84568c98b9d not found: ID does not exist"
Aug 01 00:27:43.087821 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.087799    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:43.089514 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.089491    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"]
Aug 01 00:27:43.092228 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.092211    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7qk68"]
Aug 01 00:27:43.224688 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.224660    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"]
Aug 01 00:27:43.226718 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:27:43.226694    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b07345_6b7d_478f_9e2c_6053ec761612.slice/crio-0dd82f77f9a8bf40b4eb22ff9303c994922caddafaa4d1b3d0d92715219b4bce WatchSource:0}: Error finding container 0dd82f77f9a8bf40b4eb22ff9303c994922caddafaa4d1b3d0d92715219b4bce: Status 404 returned error can't find the container with id 0dd82f77f9a8bf40b4eb22ff9303c994922caddafaa4d1b3d0d92715219b4bce
Aug 01 00:27:43.281509 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.281484    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c88335-09a1-4426-abaa-ffa022d45366-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:43.283675 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.283651    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c88335-09a1-4426-abaa-ffa022d45366-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:43.526344 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.526259    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:43.669848 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:43.669821    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"]
Aug 01 00:27:43.673088 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:27:43.673058    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c88335_09a1_4426_abaa_ffa022d45366.slice/crio-52bf9d71489765d8d0389456e5dea124768e00a5eeb855ff9409bc68770b6a54 WatchSource:0}: Error finding container 52bf9d71489765d8d0389456e5dea124768e00a5eeb855ff9409bc68770b6a54: Status 404 returned error can't find the container with id 52bf9d71489765d8d0389456e5dea124768e00a5eeb855ff9409bc68770b6a54
Aug 01 00:27:44.066182 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:44.066145    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv" event={"ID":"62c88335-09a1-4426-abaa-ffa022d45366","Type":"ContainerStarted","Data":"309d0c73cf998efdf56840ffa4298295429687be752af5c97b9af4435e853584"}
Aug 01 00:27:44.066408 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:44.066203    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv" event={"ID":"62c88335-09a1-4426-abaa-ffa022d45366","Type":"ContainerStarted","Data":"52bf9d71489765d8d0389456e5dea124768e00a5eeb855ff9409bc68770b6a54"}
Aug 01 00:27:44.067745 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:44.067722    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" event={"ID":"85b07345-6b7d-478f-9e2c-6053ec761612","Type":"ContainerStarted","Data":"b7046aba6b40a6f6663bc2bdccde92d365fd9bbe69df0b6a8d5b7d499b37c732"}
Aug 01 00:27:44.067851 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:44.067765    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" event={"ID":"85b07345-6b7d-478f-9e2c-6053ec761612","Type":"ContainerStarted","Data":"0dd82f77f9a8bf40b4eb22ff9303c994922caddafaa4d1b3d0d92715219b4bce"}
Aug 01 00:27:44.074841 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:44.074817    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87484a9f-8b31-45b3-b641-79043251ee9d" path="/var/lib/kubelet/pods/87484a9f-8b31-45b3-b641-79043251ee9d/volumes"
Aug 01 00:27:48.085512 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:48.085421    2546 generic.go:356] "Generic (PLEG): container finished" podID="62c88335-09a1-4426-abaa-ffa022d45366" containerID="309d0c73cf998efdf56840ffa4298295429687be752af5c97b9af4435e853584" exitCode=0
Aug 01 00:27:48.085512 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:48.085497    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv" event={"ID":"62c88335-09a1-4426-abaa-ffa022d45366","Type":"ContainerDied","Data":"309d0c73cf998efdf56840ffa4298295429687be752af5c97b9af4435e853584"}
Aug 01 00:27:48.086939 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:48.086917    2546 generic.go:356] "Generic (PLEG): container finished" podID="85b07345-6b7d-478f-9e2c-6053ec761612" containerID="b7046aba6b40a6f6663bc2bdccde92d365fd9bbe69df0b6a8d5b7d499b37c732" exitCode=0
Aug 01 00:27:48.087019 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:48.086958    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" event={"ID":"85b07345-6b7d-478f-9e2c-6053ec761612","Type":"ContainerDied","Data":"b7046aba6b40a6f6663bc2bdccde92d365fd9bbe69df0b6a8d5b7d499b37c732"}
Aug 01 00:27:49.094266 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:49.094214    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv" event={"ID":"62c88335-09a1-4426-abaa-ffa022d45366","Type":"ContainerStarted","Data":"f1814d26263cb06d1898fe45368cf14069c61936e368e0886c0c9289323434d4"}
Aug 01 00:27:49.094713 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:49.094309    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv" event={"ID":"62c88335-09a1-4426-abaa-ffa022d45366","Type":"ContainerStarted","Data":"e29ab52034fc9de19e9cb48972b0e0050676098504498baca8520ac0051fe8a6"}
Aug 01 00:27:49.094713 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:49.094506    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:49.096310 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:49.096286    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" event={"ID":"85b07345-6b7d-478f-9e2c-6053ec761612","Type":"ContainerStarted","Data":"b68d132f5fdc0c2048f5760c088651a98ec45126d189e1658a096c5ec11490bc"}
Aug 01 00:27:49.096421 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:49.096334    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" event={"ID":"85b07345-6b7d-478f-9e2c-6053ec761612","Type":"ContainerStarted","Data":"97c0c31617e8a44137c222ab486c981d36d6ee6fbf36711490258f9832476e9a"}
Aug 01 00:27:49.096658 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:49.096626    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:49.096767 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:49.096672    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:49.098336 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:49.098313    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Aug 01 00:27:49.118999 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:49.118949    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv" podStartSLOduration=3.235245104 podStartE2EDuration="7.118931517s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.883686413s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:27:49.117684045 +0000 UTC m=+2499.577146858" watchObservedRunningTime="2026-08-01 00:27:49.118931517 +0000 UTC m=+2499.578394329"
Aug 01 00:27:49.138764 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:49.138706    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" podStartSLOduration=4.345400214 podStartE2EDuration="8.138688862s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.793288648s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:27:49.136885328 +0000 UTC m=+2499.596348143" watchObservedRunningTime="2026-08-01 00:27:49.138688862 +0000 UTC m=+2499.598151675"
Aug 01 00:27:50.101633 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:50.101594    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:27:50.102030 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:50.101689    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Aug 01 00:27:55.107288 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:55.107237    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:27:55.108321 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:55.108283    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Aug 01 00:27:56.110468 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:27:56.110438    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:28:05.108548 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:05.108458    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Aug 01 00:28:15.108600 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:15.108560    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Aug 01 00:28:25.108746 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:25.108700    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Aug 01 00:28:26.114233 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:26.114204    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:28:26.836315 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:26.836284    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:32.703699 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:32.703669    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"]
Aug 01 00:28:32.704111 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:32.703906    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv" podUID="62c88335-09a1-4426-abaa-ffa022d45366" containerName="kserve-container" containerID="cri-o://e29ab52034fc9de19e9cb48972b0e0050676098504498baca8520ac0051fe8a6" gracePeriod=30
Aug 01 00:28:32.704111 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:32.703931    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv" podUID="62c88335-09a1-4426-abaa-ffa022d45366" containerName="kube-rbac-proxy" containerID="cri-o://f1814d26263cb06d1898fe45368cf14069c61936e368e0886c0c9289323434d4" gracePeriod=30
Aug 01 00:28:32.724603 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:32.724556    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:33.279333 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.279299    2546 generic.go:356] "Generic (PLEG): container finished" podID="62c88335-09a1-4426-abaa-ffa022d45366" containerID="f1814d26263cb06d1898fe45368cf14069c61936e368e0886c0c9289323434d4" exitCode=2
Aug 01 00:28:33.279502 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.279374    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv" event={"ID":"62c88335-09a1-4426-abaa-ffa022d45366","Type":"ContainerDied","Data":"f1814d26263cb06d1898fe45368cf14069c61936e368e0886c0c9289323434d4"}
Aug 01 00:28:33.656802 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.656776    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:28:33.705762 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.705731    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2sxh\" (UniqueName: \"kubernetes.io/projected/62c88335-09a1-4426-abaa-ffa022d45366-kube-api-access-n2sxh\") pod \"62c88335-09a1-4426-abaa-ffa022d45366\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") "
Aug 01 00:28:33.706135 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.705793    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c88335-09a1-4426-abaa-ffa022d45366-proxy-tls\") pod \"62c88335-09a1-4426-abaa-ffa022d45366\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") "
Aug 01 00:28:33.706135 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.705850    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62c88335-09a1-4426-abaa-ffa022d45366-cabundle-cert\") pod \"62c88335-09a1-4426-abaa-ffa022d45366\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") "
Aug 01 00:28:33.706135 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.705874    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62c88335-09a1-4426-abaa-ffa022d45366-kube-rbac-proxy-sar-config\") pod \"62c88335-09a1-4426-abaa-ffa022d45366\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") "
Aug 01 00:28:33.706135 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.705893    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62c88335-09a1-4426-abaa-ffa022d45366-kserve-provision-location\") pod \"62c88335-09a1-4426-abaa-ffa022d45366\" (UID: \"62c88335-09a1-4426-abaa-ffa022d45366\") "
Aug 01 00:28:33.706354 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.706215    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c88335-09a1-4426-abaa-ffa022d45366-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "62c88335-09a1-4426-abaa-ffa022d45366" (UID: "62c88335-09a1-4426-abaa-ffa022d45366"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:28:33.706354 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.706223    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c88335-09a1-4426-abaa-ffa022d45366-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "62c88335-09a1-4426-abaa-ffa022d45366" (UID: "62c88335-09a1-4426-abaa-ffa022d45366"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:28:33.706354 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.706233    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c88335-09a1-4426-abaa-ffa022d45366-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62c88335-09a1-4426-abaa-ffa022d45366" (UID: "62c88335-09a1-4426-abaa-ffa022d45366"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:28:33.707759 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.707740    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c88335-09a1-4426-abaa-ffa022d45366-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "62c88335-09a1-4426-abaa-ffa022d45366" (UID: "62c88335-09a1-4426-abaa-ffa022d45366"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:28:33.707882 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.707861    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c88335-09a1-4426-abaa-ffa022d45366-kube-api-access-n2sxh" (OuterVolumeSpecName: "kube-api-access-n2sxh") pod "62c88335-09a1-4426-abaa-ffa022d45366" (UID: "62c88335-09a1-4426-abaa-ffa022d45366"). InnerVolumeSpecName "kube-api-access-n2sxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:28:33.806561 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.806529    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62c88335-09a1-4426-abaa-ffa022d45366-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:28:33.806561 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.806554    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62c88335-09a1-4426-abaa-ffa022d45366-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:28:33.806561 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.806563    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62c88335-09a1-4426-abaa-ffa022d45366-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:28:33.806772 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.806573    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2sxh\" (UniqueName: \"kubernetes.io/projected/62c88335-09a1-4426-abaa-ffa022d45366-kube-api-access-n2sxh\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:28:33.806772 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:33.806582    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62c88335-09a1-4426-abaa-ffa022d45366-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:28:34.285165 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:34.285119    2546 generic.go:356] "Generic (PLEG): container finished" podID="62c88335-09a1-4426-abaa-ffa022d45366" containerID="e29ab52034fc9de19e9cb48972b0e0050676098504498baca8520ac0051fe8a6" exitCode=0
Aug 01 00:28:34.285374 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:34.285190    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"
Aug 01 00:28:34.285374 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:34.285202    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv" event={"ID":"62c88335-09a1-4426-abaa-ffa022d45366","Type":"ContainerDied","Data":"e29ab52034fc9de19e9cb48972b0e0050676098504498baca8520ac0051fe8a6"}
Aug 01 00:28:34.285374 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:34.285299    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv" event={"ID":"62c88335-09a1-4426-abaa-ffa022d45366","Type":"ContainerDied","Data":"52bf9d71489765d8d0389456e5dea124768e00a5eeb855ff9409bc68770b6a54"}
Aug 01 00:28:34.285374 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:34.285342    2546 scope.go:117] "RemoveContainer" containerID="f1814d26263cb06d1898fe45368cf14069c61936e368e0886c0c9289323434d4"
Aug 01 00:28:34.293663 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:34.293644    2546 scope.go:117] "RemoveContainer" containerID="e29ab52034fc9de19e9cb48972b0e0050676098504498baca8520ac0051fe8a6"
Aug 01 00:28:34.301498 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:34.301470    2546 scope.go:117] "RemoveContainer" containerID="309d0c73cf998efdf56840ffa4298295429687be752af5c97b9af4435e853584"
Aug 01 00:28:34.304019 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:34.303997    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"]
Aug 01 00:28:34.306445 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:34.306410    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-4w4tv"]
Aug 01 00:28:34.309178 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:34.309156    2546 scope.go:117] "RemoveContainer" containerID="f1814d26263cb06d1898fe45368cf14069c61936e368e0886c0c9289323434d4"
Aug 01 00:28:34.309432 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:28:34.309412    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1814d26263cb06d1898fe45368cf14069c61936e368e0886c0c9289323434d4\": container with ID starting with f1814d26263cb06d1898fe45368cf14069c61936e368e0886c0c9289323434d4 not found: ID does not exist" containerID="f1814d26263cb06d1898fe45368cf14069c61936e368e0886c0c9289323434d4"
Aug 01 00:28:34.309487 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:34.309441    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1814d26263cb06d1898fe45368cf14069c61936e368e0886c0c9289323434d4"} err="failed to get container status \"f1814d26263cb06d1898fe45368cf14069c61936e368e0886c0c9289323434d4\": rpc error: code = NotFound desc = could not find container \"f1814d26263cb06d1898fe45368cf14069c61936e368e0886c0c9289323434d4\": container with ID starting with f1814d26263cb06d1898fe45368cf14069c61936e368e0886c0c9289323434d4 not found: ID does not exist"
Aug 01 00:28:34.309487 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:34.309458    2546 scope.go:117] "RemoveContainer" containerID="e29ab52034fc9de19e9cb48972b0e0050676098504498baca8520ac0051fe8a6"
Aug 01 00:28:34.309726 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:28:34.309710    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e29ab52034fc9de19e9cb48972b0e0050676098504498baca8520ac0051fe8a6\": container with ID starting with e29ab52034fc9de19e9cb48972b0e0050676098504498baca8520ac0051fe8a6 not found: ID does not exist" containerID="e29ab52034fc9de19e9cb48972b0e0050676098504498baca8520ac0051fe8a6"
Aug 01 00:28:34.309776 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:34.309729    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e29ab52034fc9de19e9cb48972b0e0050676098504498baca8520ac0051fe8a6"} err="failed to get container status \"e29ab52034fc9de19e9cb48972b0e0050676098504498baca8520ac0051fe8a6\": rpc error: code = NotFound desc = could not find container \"e29ab52034fc9de19e9cb48972b0e0050676098504498baca8520ac0051fe8a6\": container with ID starting with e29ab52034fc9de19e9cb48972b0e0050676098504498baca8520ac0051fe8a6 not found: ID does not exist"
Aug 01 00:28:34.309776 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:34.309745    2546 scope.go:117] "RemoveContainer" containerID="309d0c73cf998efdf56840ffa4298295429687be752af5c97b9af4435e853584"
Aug 01 00:28:34.309997 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:28:34.309981    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309d0c73cf998efdf56840ffa4298295429687be752af5c97b9af4435e853584\": container with ID starting with 309d0c73cf998efdf56840ffa4298295429687be752af5c97b9af4435e853584 not found: ID does not exist" containerID="309d0c73cf998efdf56840ffa4298295429687be752af5c97b9af4435e853584"
Aug 01 00:28:34.310039 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:34.310002    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309d0c73cf998efdf56840ffa4298295429687be752af5c97b9af4435e853584"} err="failed to get container status \"309d0c73cf998efdf56840ffa4298295429687be752af5c97b9af4435e853584\": rpc error: code = NotFound desc = could not find container \"309d0c73cf998efdf56840ffa4298295429687be752af5c97b9af4435e853584\": container with ID starting with 309d0c73cf998efdf56840ffa4298295429687be752af5c97b9af4435e853584 not found: ID does not exist"
Aug 01 00:28:35.109166 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:35.109130    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Aug 01 00:28:36.075919 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:36.075882    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62c88335-09a1-4426-abaa-ffa022d45366" path="/var/lib/kubelet/pods/62c88335-09a1-4426-abaa-ffa022d45366/volumes"
Aug 01 00:28:37.879440 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:37.879406    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:38.234124 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:38.234048    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:28:45.108715 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:45.108675    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Aug 01 00:28:55.109132 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:28:55.109101    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:29:01.384341 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:01.384306    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"]
Aug 01 00:29:01.384791 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:01.384531    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kserve-container" containerID="cri-o://97c0c31617e8a44137c222ab486c981d36d6ee6fbf36711490258f9832476e9a" gracePeriod=30
Aug 01 00:29:01.384791 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:01.384550    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kube-rbac-proxy" containerID="cri-o://b68d132f5fdc0c2048f5760c088651a98ec45126d189e1658a096c5ec11490bc" gracePeriod=30
Aug 01 00:29:01.405538 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:01.405506    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:02.399479 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:02.399448    2546 generic.go:356] "Generic (PLEG): container finished" podID="85b07345-6b7d-478f-9e2c-6053ec761612" containerID="b68d132f5fdc0c2048f5760c088651a98ec45126d189e1658a096c5ec11490bc" exitCode=2
Aug 01 00:29:02.399840 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:02.399492    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" event={"ID":"85b07345-6b7d-478f-9e2c-6053ec761612","Type":"ContainerDied","Data":"b68d132f5fdc0c2048f5760c088651a98ec45126d189e1658a096c5ec11490bc"}
Aug 01 00:29:04.410023 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.409995    2546 generic.go:356] "Generic (PLEG): container finished" podID="85b07345-6b7d-478f-9e2c-6053ec761612" containerID="97c0c31617e8a44137c222ab486c981d36d6ee6fbf36711490258f9832476e9a" exitCode=0
Aug 01 00:29:04.410337 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.410064    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" event={"ID":"85b07345-6b7d-478f-9e2c-6053ec761612","Type":"ContainerDied","Data":"97c0c31617e8a44137c222ab486c981d36d6ee6fbf36711490258f9832476e9a"}
Aug 01 00:29:04.426554 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.426529    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:29:04.551300 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.551223    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85b07345-6b7d-478f-9e2c-6053ec761612-kserve-provision-location\") pod \"85b07345-6b7d-478f-9e2c-6053ec761612\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") "
Aug 01 00:29:04.551300 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.551270    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85b07345-6b7d-478f-9e2c-6053ec761612-kube-rbac-proxy-sar-config\") pod \"85b07345-6b7d-478f-9e2c-6053ec761612\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") "
Aug 01 00:29:04.551300 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.551300    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hk9br\" (UniqueName: \"kubernetes.io/projected/85b07345-6b7d-478f-9e2c-6053ec761612-kube-api-access-hk9br\") pod \"85b07345-6b7d-478f-9e2c-6053ec761612\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") "
Aug 01 00:29:04.551560 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.551350    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b07345-6b7d-478f-9e2c-6053ec761612-proxy-tls\") pod \"85b07345-6b7d-478f-9e2c-6053ec761612\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") "
Aug 01 00:29:04.551560 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.551413    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85b07345-6b7d-478f-9e2c-6053ec761612-cabundle-cert\") pod \"85b07345-6b7d-478f-9e2c-6053ec761612\" (UID: \"85b07345-6b7d-478f-9e2c-6053ec761612\") "
Aug 01 00:29:04.551560 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.551525    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b07345-6b7d-478f-9e2c-6053ec761612-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "85b07345-6b7d-478f-9e2c-6053ec761612" (UID: "85b07345-6b7d-478f-9e2c-6053ec761612"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:29:04.551712 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.551625    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b07345-6b7d-478f-9e2c-6053ec761612-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "85b07345-6b7d-478f-9e2c-6053ec761612" (UID: "85b07345-6b7d-478f-9e2c-6053ec761612"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:29:04.551712 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.551654    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85b07345-6b7d-478f-9e2c-6053ec761612-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:29:04.551835 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.551816    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b07345-6b7d-478f-9e2c-6053ec761612-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "85b07345-6b7d-478f-9e2c-6053ec761612" (UID: "85b07345-6b7d-478f-9e2c-6053ec761612"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:29:04.553349 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.553328    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85b07345-6b7d-478f-9e2c-6053ec761612-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85b07345-6b7d-478f-9e2c-6053ec761612" (UID: "85b07345-6b7d-478f-9e2c-6053ec761612"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:29:04.553455 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.553384    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b07345-6b7d-478f-9e2c-6053ec761612-kube-api-access-hk9br" (OuterVolumeSpecName: "kube-api-access-hk9br") pod "85b07345-6b7d-478f-9e2c-6053ec761612" (UID: "85b07345-6b7d-478f-9e2c-6053ec761612"). InnerVolumeSpecName "kube-api-access-hk9br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:29:04.652017 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.651981    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85b07345-6b7d-478f-9e2c-6053ec761612-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:29:04.652017 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.652013    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85b07345-6b7d-478f-9e2c-6053ec761612-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:29:04.652017 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.652024    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk9br\" (UniqueName: \"kubernetes.io/projected/85b07345-6b7d-478f-9e2c-6053ec761612-kube-api-access-hk9br\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:29:04.652191 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:04.652034    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b07345-6b7d-478f-9e2c-6053ec761612-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:29:05.415283 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:05.415238    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp" event={"ID":"85b07345-6b7d-478f-9e2c-6053ec761612","Type":"ContainerDied","Data":"0dd82f77f9a8bf40b4eb22ff9303c994922caddafaa4d1b3d0d92715219b4bce"}
Aug 01 00:29:05.415675 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:05.415295    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"
Aug 01 00:29:05.415675 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:05.415313    2546 scope.go:117] "RemoveContainer" containerID="b68d132f5fdc0c2048f5760c088651a98ec45126d189e1658a096c5ec11490bc"
Aug 01 00:29:05.423920 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:05.423903    2546 scope.go:117] "RemoveContainer" containerID="97c0c31617e8a44137c222ab486c981d36d6ee6fbf36711490258f9832476e9a"
Aug 01 00:29:05.430930 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:05.430914    2546 scope.go:117] "RemoveContainer" containerID="b7046aba6b40a6f6663bc2bdccde92d365fd9bbe69df0b6a8d5b7d499b37c732"
Aug 01 00:29:05.436558 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:05.436537    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"]
Aug 01 00:29:05.442474 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:05.442451    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-klxcp"]
Aug 01 00:29:06.075345 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.075310    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" path="/var/lib/kubelet/pods/85b07345-6b7d-478f-9e2c-6053ec761612/volumes"
Aug 01 00:29:06.558240 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.558208    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:06.590824 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.590794    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"]
Aug 01 00:29:06.591113 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591101    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62c88335-09a1-4426-abaa-ffa022d45366" containerName="storage-initializer"
Aug 01 00:29:06.591166 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591114    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="62c88335-09a1-4426-abaa-ffa022d45366" containerName="storage-initializer"
Aug 01 00:29:06.591166 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591129    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kube-rbac-proxy"
Aug 01 00:29:06.591166 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591135    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kube-rbac-proxy"
Aug 01 00:29:06.591166 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591145    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kserve-container"
Aug 01 00:29:06.591166 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591150    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kserve-container"
Aug 01 00:29:06.591166 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591157    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62c88335-09a1-4426-abaa-ffa022d45366" containerName="kube-rbac-proxy"
Aug 01 00:29:06.591166 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591162    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="62c88335-09a1-4426-abaa-ffa022d45366" containerName="kube-rbac-proxy"
Aug 01 00:29:06.591437 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591172    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="storage-initializer"
Aug 01 00:29:06.591437 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591178    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="storage-initializer"
Aug 01 00:29:06.591437 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591188    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62c88335-09a1-4426-abaa-ffa022d45366" containerName="kserve-container"
Aug 01 00:29:06.591437 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591194    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="62c88335-09a1-4426-abaa-ffa022d45366" containerName="kserve-container"
Aug 01 00:29:06.591437 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591238    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kube-rbac-proxy"
Aug 01 00:29:06.591437 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591245    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="85b07345-6b7d-478f-9e2c-6053ec761612" containerName="kserve-container"
Aug 01 00:29:06.591437 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591264    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="62c88335-09a1-4426-abaa-ffa022d45366" containerName="kserve-container"
Aug 01 00:29:06.591437 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.591270    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="62c88335-09a1-4426-abaa-ffa022d45366" containerName="kube-rbac-proxy"
Aug 01 00:29:06.595917 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.595902    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.598590 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.598570    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Aug 01 00:29:06.598722 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.598684    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:29:06.598722 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.598684    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:29:06.598904 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.598719    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:29:06.599661 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.599640    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:29:06.599749 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.599671    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:29:06.599749 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.599716    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Aug 01 00:29:06.612053 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.612034    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"]
Aug 01 00:29:06.766674 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.766645    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3c48cbe-fa51-4022-ad87-1016d775637b-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.766788 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.766691    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3c48cbe-fa51-4022-ad87-1016d775637b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.766788 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.766712    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49tk\" (UniqueName: \"kubernetes.io/projected/c3c48cbe-fa51-4022-ad87-1016d775637b-kube-api-access-j49tk\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.766879 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.766782    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3c48cbe-fa51-4022-ad87-1016d775637b-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.766879 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.766820    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3c48cbe-fa51-4022-ad87-1016d775637b-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.867503 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.867483    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3c48cbe-fa51-4022-ad87-1016d775637b-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.867654 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.867526    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3c48cbe-fa51-4022-ad87-1016d775637b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.867654 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.867542    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j49tk\" (UniqueName: \"kubernetes.io/projected/c3c48cbe-fa51-4022-ad87-1016d775637b-kube-api-access-j49tk\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.867654 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.867571    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3c48cbe-fa51-4022-ad87-1016d775637b-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.867818 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.867668    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3c48cbe-fa51-4022-ad87-1016d775637b-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.867906 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.867882    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3c48cbe-fa51-4022-ad87-1016d775637b-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.868244 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.868223    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3c48cbe-fa51-4022-ad87-1016d775637b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.868244 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.868239    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3c48cbe-fa51-4022-ad87-1016d775637b-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.869871 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.869851    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3c48cbe-fa51-4022-ad87-1016d775637b-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.891142 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.891117    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49tk\" (UniqueName: \"kubernetes.io/projected/c3c48cbe-fa51-4022-ad87-1016d775637b-kube-api-access-j49tk\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.906075 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.906053    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:06.967711 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:06.967679    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:07.033171 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:07.033146    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"]
Aug 01 00:29:07.040864 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:29:07.035493    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c48cbe_fa51_4022_ad87_1016d775637b.slice/crio-de4e45f1d5374ea8b146a9cda29289ecb47f85512e86c06e1651cc11af6f7719 WatchSource:0}: Error finding container de4e45f1d5374ea8b146a9cda29289ecb47f85512e86c06e1651cc11af6f7719: Status 404 returned error can't find the container with id de4e45f1d5374ea8b146a9cda29289ecb47f85512e86c06e1651cc11af6f7719
Aug 01 00:29:07.425530 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:07.425494    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh" event={"ID":"c3c48cbe-fa51-4022-ad87-1016d775637b","Type":"ContainerStarted","Data":"53ea1b785926abfcfce0aecd5c6f449f377501586b86c7d4549c552b749dda96"}
Aug 01 00:29:07.425720 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:07.425583    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh" event={"ID":"c3c48cbe-fa51-4022-ad87-1016d775637b","Type":"ContainerStarted","Data":"de4e45f1d5374ea8b146a9cda29289ecb47f85512e86c06e1651cc11af6f7719"}
Aug 01 00:29:08.000376 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:08.000344    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:11.443014 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:11.442936    2546 generic.go:356] "Generic (PLEG): container finished" podID="c3c48cbe-fa51-4022-ad87-1016d775637b" containerID="53ea1b785926abfcfce0aecd5c6f449f377501586b86c7d4549c552b749dda96" exitCode=0
Aug 01 00:29:11.443014 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:11.442982    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh" event={"ID":"c3c48cbe-fa51-4022-ad87-1016d775637b","Type":"ContainerDied","Data":"53ea1b785926abfcfce0aecd5c6f449f377501586b86c7d4549c552b749dda96"}
Aug 01 00:29:12.449281 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:12.449236    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh" event={"ID":"c3c48cbe-fa51-4022-ad87-1016d775637b","Type":"ContainerStarted","Data":"e081c691313d46b0440493c44f6bc54027a2b2ce1f56bdc256cf9b9a1da17c54"}
Aug 01 00:29:12.449637 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:12.449313    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh" event={"ID":"c3c48cbe-fa51-4022-ad87-1016d775637b","Type":"ContainerStarted","Data":"e5403ba220446e89474cabc94f82df3ae8e4a956962f2a8eac22c7d89e26712c"}
Aug 01 00:29:12.449637 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:12.449501    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:12.468756 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:12.468713    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh" podStartSLOduration=2.393213818 podStartE2EDuration="6.468698007s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.075484189s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:12.467888539 +0000 UTC m=+2582.927351352" watchObservedRunningTime="2026-08-01 00:29:12.468698007 +0000 UTC m=+2582.928160820"
Aug 01 00:29:13.127750 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.127710    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:13.158567 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.158536    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"]
Aug 01 00:29:13.161576 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.161559    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.163942 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.163921    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Aug 01 00:29:13.164063 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.164027    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Aug 01 00:29:13.172080 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.172059    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"]
Aug 01 00:29:13.318244 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.318212    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc4bp\" (UniqueName: \"kubernetes.io/projected/58b2357f-928d-4103-9786-e4c161f51ae8-kube-api-access-cc4bp\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.318244 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.318246    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58b2357f-928d-4103-9786-e4c161f51ae8-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.318244 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.318351    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58b2357f-928d-4103-9786-e4c161f51ae8-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.318578 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.318396    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58b2357f-928d-4103-9786-e4c161f51ae8-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.318578 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.318416    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58b2357f-928d-4103-9786-e4c161f51ae8-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.419705 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.419640    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58b2357f-928d-4103-9786-e4c161f51ae8-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.419705 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.419685    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58b2357f-928d-4103-9786-e4c161f51ae8-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.419899 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.419709    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58b2357f-928d-4103-9786-e4c161f51ae8-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.419899 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.419736    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc4bp\" (UniqueName: \"kubernetes.io/projected/58b2357f-928d-4103-9786-e4c161f51ae8-kube-api-access-cc4bp\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.419899 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.419766    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58b2357f-928d-4103-9786-e4c161f51ae8-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.419899 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:29:13.419882    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Aug 01 00:29:13.420122 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:29:13.419952    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58b2357f-928d-4103-9786-e4c161f51ae8-proxy-tls podName:58b2357f-928d-4103-9786-e4c161f51ae8 nodeName:}" failed. No retries permitted until 2026-08-01 00:29:13.919933069 +0000 UTC m=+2584.379395862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/58b2357f-928d-4103-9786-e4c161f51ae8-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf" (UID: "58b2357f-928d-4103-9786-e4c161f51ae8") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Aug 01 00:29:13.420206 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.420186    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58b2357f-928d-4103-9786-e4c161f51ae8-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.420406 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.420387    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58b2357f-928d-4103-9786-e4c161f51ae8-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.420500 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.420478    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58b2357f-928d-4103-9786-e4c161f51ae8-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.428047 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.428016    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc4bp\" (UniqueName: \"kubernetes.io/projected/58b2357f-928d-4103-9786-e4c161f51ae8-kube-api-access-cc4bp\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.455409 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.455384    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:13.923914 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.923882    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58b2357f-928d-4103-9786-e4c161f51ae8-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:13.926244 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:13.926216    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58b2357f-928d-4103-9786-e4c161f51ae8-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:14.071542 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:14.071505    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:14.197888 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:14.197863    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"]
Aug 01 00:29:14.199928 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:29:14.199894    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b2357f_928d_4103_9786_e4c161f51ae8.slice/crio-959b5567c7c5220d8f10b1968742ad18292a4f71bb02c528a97199c982a379cd WatchSource:0}: Error finding container 959b5567c7c5220d8f10b1968742ad18292a4f71bb02c528a97199c982a379cd: Status 404 returned error can't find the container with id 959b5567c7c5220d8f10b1968742ad18292a4f71bb02c528a97199c982a379cd
Aug 01 00:29:14.460760 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:14.460674    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf" event={"ID":"58b2357f-928d-4103-9786-e4c161f51ae8","Type":"ContainerStarted","Data":"fb87f48e90f9a6334b5c17a8f82ab40a71b474e88c86e1ab5a04b41480d997b2"}
Aug 01 00:29:14.460760 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:14.460748    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf" event={"ID":"58b2357f-928d-4103-9786-e4c161f51ae8","Type":"ContainerStarted","Data":"959b5567c7c5220d8f10b1968742ad18292a4f71bb02c528a97199c982a379cd"}
Aug 01 00:29:14.488414 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:14.488359    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:18.479419 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:18.479338    2546 generic.go:356] "Generic (PLEG): container finished" podID="58b2357f-928d-4103-9786-e4c161f51ae8" containerID="fb87f48e90f9a6334b5c17a8f82ab40a71b474e88c86e1ab5a04b41480d997b2" exitCode=0
Aug 01 00:29:18.479419 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:18.479410    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf" event={"ID":"58b2357f-928d-4103-9786-e4c161f51ae8","Type":"ContainerDied","Data":"fb87f48e90f9a6334b5c17a8f82ab40a71b474e88c86e1ab5a04b41480d997b2"}
Aug 01 00:29:19.466909 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:19.466880    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:29:19.487764 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:19.487733    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf" event={"ID":"58b2357f-928d-4103-9786-e4c161f51ae8","Type":"ContainerStarted","Data":"54150941630a5e89deec25391006c7275fa7129d9e2e083e0223584d8885c8dc"}
Aug 01 00:29:19.488142 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:19.487787    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf" event={"ID":"58b2357f-928d-4103-9786-e4c161f51ae8","Type":"ContainerStarted","Data":"0806e98737dcb3fe617c194bd85722cd1e9d4c143740d7aef85144a3c196d931"}
Aug 01 00:29:19.488142 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:19.487994    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:19.488238 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:19.488158    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:19.505889 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:19.505848    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf" podStartSLOduration=2.531315122 podStartE2EDuration="6.505834638s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.974519516s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:29:19.504759428 +0000 UTC m=+2589.964222240" watchObservedRunningTime="2026-08-01 00:29:19.505834638 +0000 UTC m=+2589.965297450"
Aug 01 00:29:22.108787 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:22.108760    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:22.170743 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:22.170714    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:25.497207 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:25.497180    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:40.684569 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:40.684533    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:29:49.520555 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:49.520508    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh" podUID="c3c48cbe-fa51-4022-ad87-1016d775637b" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Aug 01 00:29:55.500833 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:55.500802    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:29:59.470050 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:29:59.470021    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:30:03.256862 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:03.256821    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"]
Aug 01 00:30:03.257369 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:03.257075    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf" podUID="58b2357f-928d-4103-9786-e4c161f51ae8" containerName="kserve-container" containerID="cri-o://0806e98737dcb3fe617c194bd85722cd1e9d4c143740d7aef85144a3c196d931" gracePeriod=30
Aug 01 00:30:03.257369 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:03.257089    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf" podUID="58b2357f-928d-4103-9786-e4c161f51ae8" containerName="kube-rbac-proxy" containerID="cri-o://54150941630a5e89deec25391006c7275fa7129d9e2e083e0223584d8885c8dc" gracePeriod=30
Aug 01 00:30:03.276320 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:03.276295    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:03.678241 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:03.678213    2546 generic.go:356] "Generic (PLEG): container finished" podID="58b2357f-928d-4103-9786-e4c161f51ae8" containerID="54150941630a5e89deec25391006c7275fa7129d9e2e083e0223584d8885c8dc" exitCode=2
Aug 01 00:30:03.678411 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:03.678289    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf" event={"ID":"58b2357f-928d-4103-9786-e4c161f51ae8","Type":"ContainerDied","Data":"54150941630a5e89deec25391006c7275fa7129d9e2e083e0223584d8885c8dc"}
Aug 01 00:30:04.397283 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.397244    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:30:04.533869 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.533789    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58b2357f-928d-4103-9786-e4c161f51ae8-cabundle-cert\") pod \"58b2357f-928d-4103-9786-e4c161f51ae8\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") "
Aug 01 00:30:04.533869 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.533830    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58b2357f-928d-4103-9786-e4c161f51ae8-kserve-provision-location\") pod \"58b2357f-928d-4103-9786-e4c161f51ae8\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") "
Aug 01 00:30:04.533869 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.533859    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58b2357f-928d-4103-9786-e4c161f51ae8-kube-rbac-proxy-sar-config\") pod \"58b2357f-928d-4103-9786-e4c161f51ae8\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") "
Aug 01 00:30:04.534158 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.533934    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58b2357f-928d-4103-9786-e4c161f51ae8-proxy-tls\") pod \"58b2357f-928d-4103-9786-e4c161f51ae8\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") "
Aug 01 00:30:04.534158 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.533959    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc4bp\" (UniqueName: \"kubernetes.io/projected/58b2357f-928d-4103-9786-e4c161f51ae8-kube-api-access-cc4bp\") pod \"58b2357f-928d-4103-9786-e4c161f51ae8\" (UID: \"58b2357f-928d-4103-9786-e4c161f51ae8\") "
Aug 01 00:30:04.534279 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.534171    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b2357f-928d-4103-9786-e4c161f51ae8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "58b2357f-928d-4103-9786-e4c161f51ae8" (UID: "58b2357f-928d-4103-9786-e4c161f51ae8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:30:04.534279 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.534182    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b2357f-928d-4103-9786-e4c161f51ae8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "58b2357f-928d-4103-9786-e4c161f51ae8" (UID: "58b2357f-928d-4103-9786-e4c161f51ae8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:30:04.534279 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.534184    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b2357f-928d-4103-9786-e4c161f51ae8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "58b2357f-928d-4103-9786-e4c161f51ae8" (UID: "58b2357f-928d-4103-9786-e4c161f51ae8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:30:04.535908 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.535884    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58b2357f-928d-4103-9786-e4c161f51ae8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "58b2357f-928d-4103-9786-e4c161f51ae8" (UID: "58b2357f-928d-4103-9786-e4c161f51ae8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:30:04.536045 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.536024    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b2357f-928d-4103-9786-e4c161f51ae8-kube-api-access-cc4bp" (OuterVolumeSpecName: "kube-api-access-cc4bp") pod "58b2357f-928d-4103-9786-e4c161f51ae8" (UID: "58b2357f-928d-4103-9786-e4c161f51ae8"). InnerVolumeSpecName "kube-api-access-cc4bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:30:04.634598 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.634559    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58b2357f-928d-4103-9786-e4c161f51ae8-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:30:04.634598 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.634593    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc4bp\" (UniqueName: \"kubernetes.io/projected/58b2357f-928d-4103-9786-e4c161f51ae8-kube-api-access-cc4bp\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:30:04.634598 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.634603    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58b2357f-928d-4103-9786-e4c161f51ae8-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:30:04.634598 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.634612    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58b2357f-928d-4103-9786-e4c161f51ae8-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:30:04.634832 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.634620    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58b2357f-928d-4103-9786-e4c161f51ae8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:30:04.684785 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.684756    2546 generic.go:356] "Generic (PLEG): container finished" podID="58b2357f-928d-4103-9786-e4c161f51ae8" containerID="0806e98737dcb3fe617c194bd85722cd1e9d4c143740d7aef85144a3c196d931" exitCode=0
Aug 01 00:30:04.684915 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.684811    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"
Aug 01 00:30:04.684915 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.684845    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf" event={"ID":"58b2357f-928d-4103-9786-e4c161f51ae8","Type":"ContainerDied","Data":"0806e98737dcb3fe617c194bd85722cd1e9d4c143740d7aef85144a3c196d931"}
Aug 01 00:30:04.685007 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.684929    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf" event={"ID":"58b2357f-928d-4103-9786-e4c161f51ae8","Type":"ContainerDied","Data":"959b5567c7c5220d8f10b1968742ad18292a4f71bb02c528a97199c982a379cd"}
Aug 01 00:30:04.685007 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.684957    2546 scope.go:117] "RemoveContainer" containerID="54150941630a5e89deec25391006c7275fa7129d9e2e083e0223584d8885c8dc"
Aug 01 00:30:04.693897 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.693883    2546 scope.go:117] "RemoveContainer" containerID="0806e98737dcb3fe617c194bd85722cd1e9d4c143740d7aef85144a3c196d931"
Aug 01 00:30:04.701589 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.701574    2546 scope.go:117] "RemoveContainer" containerID="fb87f48e90f9a6334b5c17a8f82ab40a71b474e88c86e1ab5a04b41480d997b2"
Aug 01 00:30:04.706741 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.706716    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"]
Aug 01 00:30:04.709037 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.709023    2546 scope.go:117] "RemoveContainer" containerID="54150941630a5e89deec25391006c7275fa7129d9e2e083e0223584d8885c8dc"
Aug 01 00:30:04.709365 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:30:04.709337    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54150941630a5e89deec25391006c7275fa7129d9e2e083e0223584d8885c8dc\": container with ID starting with 54150941630a5e89deec25391006c7275fa7129d9e2e083e0223584d8885c8dc not found: ID does not exist" containerID="54150941630a5e89deec25391006c7275fa7129d9e2e083e0223584d8885c8dc"
Aug 01 00:30:04.709429 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.709372    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54150941630a5e89deec25391006c7275fa7129d9e2e083e0223584d8885c8dc"} err="failed to get container status \"54150941630a5e89deec25391006c7275fa7129d9e2e083e0223584d8885c8dc\": rpc error: code = NotFound desc = could not find container \"54150941630a5e89deec25391006c7275fa7129d9e2e083e0223584d8885c8dc\": container with ID starting with 54150941630a5e89deec25391006c7275fa7129d9e2e083e0223584d8885c8dc not found: ID does not exist"
Aug 01 00:30:04.709429 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.709390    2546 scope.go:117] "RemoveContainer" containerID="0806e98737dcb3fe617c194bd85722cd1e9d4c143740d7aef85144a3c196d931"
Aug 01 00:30:04.709622 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:30:04.709604    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0806e98737dcb3fe617c194bd85722cd1e9d4c143740d7aef85144a3c196d931\": container with ID starting with 0806e98737dcb3fe617c194bd85722cd1e9d4c143740d7aef85144a3c196d931 not found: ID does not exist" containerID="0806e98737dcb3fe617c194bd85722cd1e9d4c143740d7aef85144a3c196d931"
Aug 01 00:30:04.709696 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.709633    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0806e98737dcb3fe617c194bd85722cd1e9d4c143740d7aef85144a3c196d931"} err="failed to get container status \"0806e98737dcb3fe617c194bd85722cd1e9d4c143740d7aef85144a3c196d931\": rpc error: code = NotFound desc = could not find container \"0806e98737dcb3fe617c194bd85722cd1e9d4c143740d7aef85144a3c196d931\": container with ID starting with 0806e98737dcb3fe617c194bd85722cd1e9d4c143740d7aef85144a3c196d931 not found: ID does not exist"
Aug 01 00:30:04.709696 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.709655    2546 scope.go:117] "RemoveContainer" containerID="fb87f48e90f9a6334b5c17a8f82ab40a71b474e88c86e1ab5a04b41480d997b2"
Aug 01 00:30:04.709894 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:30:04.709875    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb87f48e90f9a6334b5c17a8f82ab40a71b474e88c86e1ab5a04b41480d997b2\": container with ID starting with fb87f48e90f9a6334b5c17a8f82ab40a71b474e88c86e1ab5a04b41480d997b2 not found: ID does not exist" containerID="fb87f48e90f9a6334b5c17a8f82ab40a71b474e88c86e1ab5a04b41480d997b2"
Aug 01 00:30:04.709939 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.709898    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb87f48e90f9a6334b5c17a8f82ab40a71b474e88c86e1ab5a04b41480d997b2"} err="failed to get container status \"fb87f48e90f9a6334b5c17a8f82ab40a71b474e88c86e1ab5a04b41480d997b2\": rpc error: code = NotFound desc = could not find container \"fb87f48e90f9a6334b5c17a8f82ab40a71b474e88c86e1ab5a04b41480d997b2\": container with ID starting with fb87f48e90f9a6334b5c17a8f82ab40a71b474e88c86e1ab5a04b41480d997b2 not found: ID does not exist"
Aug 01 00:30:04.714050 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:04.714031    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vzghf"]
Aug 01 00:30:06.075668 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:06.075636    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58b2357f-928d-4103-9786-e4c161f51ae8" path="/var/lib/kubelet/pods/58b2357f-928d-4103-9786-e4c161f51ae8/volumes"
Aug 01 00:30:06.691358 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:06.691327    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"]
Aug 01 00:30:06.691631 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:06.691603    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh" podUID="c3c48cbe-fa51-4022-ad87-1016d775637b" containerName="kserve-container" containerID="cri-o://e5403ba220446e89474cabc94f82df3ae8e4a956962f2a8eac22c7d89e26712c" gracePeriod=30
Aug 01 00:30:06.691722 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:06.691638    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh" podUID="c3c48cbe-fa51-4022-ad87-1016d775637b" containerName="kube-rbac-proxy" containerID="cri-o://e081c691313d46b0440493c44f6bc54027a2b2ce1f56bdc256cf9b9a1da17c54" gracePeriod=30
Aug 01 00:30:06.705056 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:06.705037    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:07.699987 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:07.699952    2546 generic.go:356] "Generic (PLEG): container finished" podID="c3c48cbe-fa51-4022-ad87-1016d775637b" containerID="e081c691313d46b0440493c44f6bc54027a2b2ce1f56bdc256cf9b9a1da17c54" exitCode=2
Aug 01 00:30:07.700372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:07.700020    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh" event={"ID":"c3c48cbe-fa51-4022-ad87-1016d775637b","Type":"ContainerDied","Data":"e081c691313d46b0440493c44f6bc54027a2b2ce1f56bdc256cf9b9a1da17c54"}
Aug 01 00:30:08.410468 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:08.410438    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:08.938208 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:08.938174    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:09.461483 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:09.461443    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh" podUID="c3c48cbe-fa51-4022-ad87-1016d775637b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.58:8643/healthz\": dial tcp 10.133.0.58:8643: connect: connection refused"
Aug 01 00:30:11.824876 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.824842    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:11.852977 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.852950    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"]
Aug 01 00:30:11.853292 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.853279    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58b2357f-928d-4103-9786-e4c161f51ae8" containerName="kserve-container"
Aug 01 00:30:11.853345 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.853293    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="58b2357f-928d-4103-9786-e4c161f51ae8" containerName="kserve-container"
Aug 01 00:30:11.853345 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.853303    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58b2357f-928d-4103-9786-e4c161f51ae8" containerName="kube-rbac-proxy"
Aug 01 00:30:11.853345 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.853309    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="58b2357f-928d-4103-9786-e4c161f51ae8" containerName="kube-rbac-proxy"
Aug 01 00:30:11.853345 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.853331    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58b2357f-928d-4103-9786-e4c161f51ae8" containerName="storage-initializer"
Aug 01 00:30:11.853345 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.853337    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="58b2357f-928d-4103-9786-e4c161f51ae8" containerName="storage-initializer"
Aug 01 00:30:11.853522 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.853388    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="58b2357f-928d-4103-9786-e4c161f51ae8" containerName="kube-rbac-proxy"
Aug 01 00:30:11.853522 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.853396    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="58b2357f-928d-4103-9786-e4c161f51ae8" containerName="kserve-container"
Aug 01 00:30:11.856669 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.856652    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:11.859269 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.859231    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Aug 01 00:30:11.859269 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.859229    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Aug 01 00:30:11.865848 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.865824    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"]
Aug 01 00:30:11.898389 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.898349    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djnxz\" (UniqueName: \"kubernetes.io/projected/4abbed71-c3f9-443b-bd60-b1faab1774d7-kube-api-access-djnxz\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6l7x5\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:11.898542 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.898516    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4abbed71-c3f9-443b-bd60-b1faab1774d7-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6l7x5\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:11.898614 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.898570    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4abbed71-c3f9-443b-bd60-b1faab1774d7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6l7x5\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:11.898614 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.898600    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4abbed71-c3f9-443b-bd60-b1faab1774d7-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6l7x5\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:11.898715 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.898666    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4abbed71-c3f9-443b-bd60-b1faab1774d7-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6l7x5\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:11.960410 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.960386    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:11.999391 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.999369    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4abbed71-c3f9-443b-bd60-b1faab1774d7-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6l7x5\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:11.999478 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.999406    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djnxz\" (UniqueName: \"kubernetes.io/projected/4abbed71-c3f9-443b-bd60-b1faab1774d7-kube-api-access-djnxz\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6l7x5\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:11.999478 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.999448    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4abbed71-c3f9-443b-bd60-b1faab1774d7-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6l7x5\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:11.999565 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.999475    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4abbed71-c3f9-443b-bd60-b1faab1774d7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6l7x5\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:11.999565 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.999498    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4abbed71-c3f9-443b-bd60-b1faab1774d7-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6l7x5\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:11.999820 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:11.999799    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4abbed71-c3f9-443b-bd60-b1faab1774d7-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6l7x5\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:12.000103 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:12.000086    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4abbed71-c3f9-443b-bd60-b1faab1774d7-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6l7x5\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:12.000158 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:12.000143    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4abbed71-c3f9-443b-bd60-b1faab1774d7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6l7x5\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:12.001753 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:12.001731    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4abbed71-c3f9-443b-bd60-b1faab1774d7-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6l7x5\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:12.007552 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:12.007534    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djnxz\" (UniqueName: \"kubernetes.io/projected/4abbed71-c3f9-443b-bd60-b1faab1774d7-kube-api-access-djnxz\") pod \"isvc-xgboost-v2-predictor-fcc668bc-6l7x5\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:12.166482 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:12.166435    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:12.297811 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:12.297779    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"]
Aug 01 00:30:12.300899 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:30:12.300873    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abbed71_c3f9_443b_bd60_b1faab1774d7.slice/crio-8ffa795319c574e1d05614f05d125183e7267006fe48d47faf3a39934587f593 WatchSource:0}: Error finding container 8ffa795319c574e1d05614f05d125183e7267006fe48d47faf3a39934587f593: Status 404 returned error can't find the container with id 8ffa795319c574e1d05614f05d125183e7267006fe48d47faf3a39934587f593
Aug 01 00:30:12.721136 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:12.721095    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" event={"ID":"4abbed71-c3f9-443b-bd60-b1faab1774d7","Type":"ContainerStarted","Data":"78ab2b3975d04a9a0362329673cd11730e65a812935e14d432d5ffd2771b20ac"}
Aug 01 00:30:12.721357 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:12.721163    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" event={"ID":"4abbed71-c3f9-443b-bd60-b1faab1774d7","Type":"ContainerStarted","Data":"8ffa795319c574e1d05614f05d125183e7267006fe48d47faf3a39934587f593"}
Aug 01 00:30:13.634513 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.634491    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:30:13.712482 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.712417    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j49tk\" (UniqueName: \"kubernetes.io/projected/c3c48cbe-fa51-4022-ad87-1016d775637b-kube-api-access-j49tk\") pod \"c3c48cbe-fa51-4022-ad87-1016d775637b\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") "
Aug 01 00:30:13.712482 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.712447    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3c48cbe-fa51-4022-ad87-1016d775637b-proxy-tls\") pod \"c3c48cbe-fa51-4022-ad87-1016d775637b\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") "
Aug 01 00:30:13.712482 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.712473    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3c48cbe-fa51-4022-ad87-1016d775637b-kserve-provision-location\") pod \"c3c48cbe-fa51-4022-ad87-1016d775637b\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") "
Aug 01 00:30:13.712703 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.712512    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3c48cbe-fa51-4022-ad87-1016d775637b-cabundle-cert\") pod \"c3c48cbe-fa51-4022-ad87-1016d775637b\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") "
Aug 01 00:30:13.712703 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.712536    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3c48cbe-fa51-4022-ad87-1016d775637b-kube-rbac-proxy-sar-config\") pod \"c3c48cbe-fa51-4022-ad87-1016d775637b\" (UID: \"c3c48cbe-fa51-4022-ad87-1016d775637b\") "
Aug 01 00:30:13.712820 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.712791    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c48cbe-fa51-4022-ad87-1016d775637b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c3c48cbe-fa51-4022-ad87-1016d775637b" (UID: "c3c48cbe-fa51-4022-ad87-1016d775637b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:30:13.712936 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.712905    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c48cbe-fa51-4022-ad87-1016d775637b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c3c48cbe-fa51-4022-ad87-1016d775637b" (UID: "c3c48cbe-fa51-4022-ad87-1016d775637b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:30:13.712998 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.712979    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c48cbe-fa51-4022-ad87-1016d775637b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c3c48cbe-fa51-4022-ad87-1016d775637b" (UID: "c3c48cbe-fa51-4022-ad87-1016d775637b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:30:13.714658 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.714630    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3c48cbe-fa51-4022-ad87-1016d775637b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c3c48cbe-fa51-4022-ad87-1016d775637b" (UID: "c3c48cbe-fa51-4022-ad87-1016d775637b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:30:13.714768 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.714655    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c48cbe-fa51-4022-ad87-1016d775637b-kube-api-access-j49tk" (OuterVolumeSpecName: "kube-api-access-j49tk") pod "c3c48cbe-fa51-4022-ad87-1016d775637b" (UID: "c3c48cbe-fa51-4022-ad87-1016d775637b"). InnerVolumeSpecName "kube-api-access-j49tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:30:13.727192 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.727164    2546 generic.go:356] "Generic (PLEG): container finished" podID="c3c48cbe-fa51-4022-ad87-1016d775637b" containerID="e5403ba220446e89474cabc94f82df3ae8e4a956962f2a8eac22c7d89e26712c" exitCode=0
Aug 01 00:30:13.727303 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.727216    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"
Aug 01 00:30:13.727303 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.727209    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh" event={"ID":"c3c48cbe-fa51-4022-ad87-1016d775637b","Type":"ContainerDied","Data":"e5403ba220446e89474cabc94f82df3ae8e4a956962f2a8eac22c7d89e26712c"}
Aug 01 00:30:13.727378 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.727343    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh" event={"ID":"c3c48cbe-fa51-4022-ad87-1016d775637b","Type":"ContainerDied","Data":"de4e45f1d5374ea8b146a9cda29289ecb47f85512e86c06e1651cc11af6f7719"}
Aug 01 00:30:13.727378 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.727370    2546 scope.go:117] "RemoveContainer" containerID="e081c691313d46b0440493c44f6bc54027a2b2ce1f56bdc256cf9b9a1da17c54"
Aug 01 00:30:13.735390 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.735369    2546 scope.go:117] "RemoveContainer" containerID="e5403ba220446e89474cabc94f82df3ae8e4a956962f2a8eac22c7d89e26712c"
Aug 01 00:30:13.743215 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.743198    2546 scope.go:117] "RemoveContainer" containerID="53ea1b785926abfcfce0aecd5c6f449f377501586b86c7d4549c552b749dda96"
Aug 01 00:30:13.748528 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.748506    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"]
Aug 01 00:30:13.751441 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.751424    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-5fnjh"]
Aug 01 00:30:13.751491 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.751428    2546 scope.go:117] "RemoveContainer" containerID="e081c691313d46b0440493c44f6bc54027a2b2ce1f56bdc256cf9b9a1da17c54"
Aug 01 00:30:13.751723 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:30:13.751708    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e081c691313d46b0440493c44f6bc54027a2b2ce1f56bdc256cf9b9a1da17c54\": container with ID starting with e081c691313d46b0440493c44f6bc54027a2b2ce1f56bdc256cf9b9a1da17c54 not found: ID does not exist" containerID="e081c691313d46b0440493c44f6bc54027a2b2ce1f56bdc256cf9b9a1da17c54"
Aug 01 00:30:13.751766 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.751731    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e081c691313d46b0440493c44f6bc54027a2b2ce1f56bdc256cf9b9a1da17c54"} err="failed to get container status \"e081c691313d46b0440493c44f6bc54027a2b2ce1f56bdc256cf9b9a1da17c54\": rpc error: code = NotFound desc = could not find container \"e081c691313d46b0440493c44f6bc54027a2b2ce1f56bdc256cf9b9a1da17c54\": container with ID starting with e081c691313d46b0440493c44f6bc54027a2b2ce1f56bdc256cf9b9a1da17c54 not found: ID does not exist"
Aug 01 00:30:13.751766 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.751753    2546 scope.go:117] "RemoveContainer" containerID="e5403ba220446e89474cabc94f82df3ae8e4a956962f2a8eac22c7d89e26712c"
Aug 01 00:30:13.751976 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:30:13.751957    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5403ba220446e89474cabc94f82df3ae8e4a956962f2a8eac22c7d89e26712c\": container with ID starting with e5403ba220446e89474cabc94f82df3ae8e4a956962f2a8eac22c7d89e26712c not found: ID does not exist" containerID="e5403ba220446e89474cabc94f82df3ae8e4a956962f2a8eac22c7d89e26712c"
Aug 01 00:30:13.752016 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.751983    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5403ba220446e89474cabc94f82df3ae8e4a956962f2a8eac22c7d89e26712c"} err="failed to get container status \"e5403ba220446e89474cabc94f82df3ae8e4a956962f2a8eac22c7d89e26712c\": rpc error: code = NotFound desc = could not find container \"e5403ba220446e89474cabc94f82df3ae8e4a956962f2a8eac22c7d89e26712c\": container with ID starting with e5403ba220446e89474cabc94f82df3ae8e4a956962f2a8eac22c7d89e26712c not found: ID does not exist"
Aug 01 00:30:13.752016 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.751997    2546 scope.go:117] "RemoveContainer" containerID="53ea1b785926abfcfce0aecd5c6f449f377501586b86c7d4549c552b749dda96"
Aug 01 00:30:13.752265 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:30:13.752234    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ea1b785926abfcfce0aecd5c6f449f377501586b86c7d4549c552b749dda96\": container with ID starting with 53ea1b785926abfcfce0aecd5c6f449f377501586b86c7d4549c552b749dda96 not found: ID does not exist" containerID="53ea1b785926abfcfce0aecd5c6f449f377501586b86c7d4549c552b749dda96"
Aug 01 00:30:13.752307 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.752278    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ea1b785926abfcfce0aecd5c6f449f377501586b86c7d4549c552b749dda96"} err="failed to get container status \"53ea1b785926abfcfce0aecd5c6f449f377501586b86c7d4549c552b749dda96\": rpc error: code = NotFound desc = could not find container \"53ea1b785926abfcfce0aecd5c6f449f377501586b86c7d4549c552b749dda96\": container with ID starting with 53ea1b785926abfcfce0aecd5c6f449f377501586b86c7d4549c552b749dda96 not found: ID does not exist"
Aug 01 00:30:13.813809 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.813776    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3c48cbe-fa51-4022-ad87-1016d775637b-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:30:13.813809 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.813808    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3c48cbe-fa51-4022-ad87-1016d775637b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:30:13.814020 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.813822    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j49tk\" (UniqueName: \"kubernetes.io/projected/c3c48cbe-fa51-4022-ad87-1016d775637b-kube-api-access-j49tk\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:30:13.814020 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.813835    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3c48cbe-fa51-4022-ad87-1016d775637b-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:30:13.814020 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:13.813848    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3c48cbe-fa51-4022-ad87-1016d775637b-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:30:14.075303 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:14.075222    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3c48cbe-fa51-4022-ad87-1016d775637b" path="/var/lib/kubelet/pods/c3c48cbe-fa51-4022-ad87-1016d775637b/volumes"
Aug 01 00:30:16.742118 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:16.742081    2546 generic.go:356] "Generic (PLEG): container finished" podID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerID="78ab2b3975d04a9a0362329673cd11730e65a812935e14d432d5ffd2771b20ac" exitCode=0
Aug 01 00:30:16.742612 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:16.742156    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" event={"ID":"4abbed71-c3f9-443b-bd60-b1faab1774d7","Type":"ContainerDied","Data":"78ab2b3975d04a9a0362329673cd11730e65a812935e14d432d5ffd2771b20ac"}
Aug 01 00:30:17.749562 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:17.749523    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" event={"ID":"4abbed71-c3f9-443b-bd60-b1faab1774d7","Type":"ContainerStarted","Data":"9b85a486266370d6bc065938cbf7f9c1d3841925ff45bc50b48ef577a0ea1a9c"}
Aug 01 00:30:17.749947 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:17.749589    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" event={"ID":"4abbed71-c3f9-443b-bd60-b1faab1774d7","Type":"ContainerStarted","Data":"1cfa9775ddd25045538f2c57ab8bfebfc6174acf8af1a40d8f18229dc3090500"}
Aug 01 00:30:17.749947 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:17.749861    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:17.750065 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:17.750008    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:17.751013 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:17.750990    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Aug 01 00:30:17.768730 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:17.768681    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" podStartSLOduration=3.087982933 podStartE2EDuration="6.768668227s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.680685294s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:30:17.766356494 +0000 UTC m=+2648.225819306" watchObservedRunningTime="2026-08-01 00:30:17.768668227 +0000 UTC m=+2648.228131039"
Aug 01 00:30:18.753640 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:18.753594    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Aug 01 00:30:23.759581 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:23.759552    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:30:23.760186 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:23.760159    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Aug 01 00:30:33.760950 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:33.760902    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Aug 01 00:30:43.215004 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:43.214973    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:30:43.760823 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:43.760788    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Aug 01 00:30:53.760130 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:30:53.760087    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Aug 01 00:31:03.760511 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:03.760434    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Aug 01 00:31:09.620411 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:09.620370    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:31:09.677156 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:09.677134    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:31:13.760910 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:13.760883    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:31:21.949694 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:21.949658    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"]
Aug 01 00:31:21.950189 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:21.949938    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kserve-container" containerID="cri-o://1cfa9775ddd25045538f2c57ab8bfebfc6174acf8af1a40d8f18229dc3090500" gracePeriod=30
Aug 01 00:31:21.950189 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:21.949970    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kube-rbac-proxy" containerID="cri-o://9b85a486266370d6bc065938cbf7f9c1d3841925ff45bc50b48ef577a0ea1a9c" gracePeriod=30
Aug 01 00:31:21.969734 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:21.969710    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:31:23.019857 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:23.019826    2546 generic.go:356] "Generic (PLEG): container finished" podID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerID="9b85a486266370d6bc065938cbf7f9c1d3841925ff45bc50b48ef577a0ea1a9c" exitCode=2
Aug 01 00:31:23.020223 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:23.019865    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" event={"ID":"4abbed71-c3f9-443b-bd60-b1faab1774d7","Type":"ContainerDied","Data":"9b85a486266370d6bc065938cbf7f9c1d3841925ff45bc50b48ef577a0ea1a9c"}
Aug 01 00:31:23.754109 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:23.754073    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Aug 01 00:31:23.761012 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:23.760990    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Aug 01 00:31:24.796867 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:24.796844    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:31:24.945408 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:24.945342    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djnxz\" (UniqueName: \"kubernetes.io/projected/4abbed71-c3f9-443b-bd60-b1faab1774d7-kube-api-access-djnxz\") pod \"4abbed71-c3f9-443b-bd60-b1faab1774d7\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") "
Aug 01 00:31:24.945408 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:24.945372    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4abbed71-c3f9-443b-bd60-b1faab1774d7-proxy-tls\") pod \"4abbed71-c3f9-443b-bd60-b1faab1774d7\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") "
Aug 01 00:31:24.945408 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:24.945409    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4abbed71-c3f9-443b-bd60-b1faab1774d7-kube-rbac-proxy-sar-config\") pod \"4abbed71-c3f9-443b-bd60-b1faab1774d7\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") "
Aug 01 00:31:24.945670 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:24.945448    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4abbed71-c3f9-443b-bd60-b1faab1774d7-kserve-provision-location\") pod \"4abbed71-c3f9-443b-bd60-b1faab1774d7\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") "
Aug 01 00:31:24.945670 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:24.945470    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4abbed71-c3f9-443b-bd60-b1faab1774d7-cabundle-cert\") pod \"4abbed71-c3f9-443b-bd60-b1faab1774d7\" (UID: \"4abbed71-c3f9-443b-bd60-b1faab1774d7\") "
Aug 01 00:31:24.945799 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:24.945772    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abbed71-c3f9-443b-bd60-b1faab1774d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4abbed71-c3f9-443b-bd60-b1faab1774d7" (UID: "4abbed71-c3f9-443b-bd60-b1faab1774d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:31:24.945834 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:24.945798    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abbed71-c3f9-443b-bd60-b1faab1774d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4abbed71-c3f9-443b-bd60-b1faab1774d7" (UID: "4abbed71-c3f9-443b-bd60-b1faab1774d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:31:24.945878 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:24.945839    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abbed71-c3f9-443b-bd60-b1faab1774d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4abbed71-c3f9-443b-bd60-b1faab1774d7" (UID: "4abbed71-c3f9-443b-bd60-b1faab1774d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:31:24.947461 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:24.947436    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4abbed71-c3f9-443b-bd60-b1faab1774d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4abbed71-c3f9-443b-bd60-b1faab1774d7" (UID: "4abbed71-c3f9-443b-bd60-b1faab1774d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:31:24.947461 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:24.947445    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abbed71-c3f9-443b-bd60-b1faab1774d7-kube-api-access-djnxz" (OuterVolumeSpecName: "kube-api-access-djnxz") pod "4abbed71-c3f9-443b-bd60-b1faab1774d7" (UID: "4abbed71-c3f9-443b-bd60-b1faab1774d7"). InnerVolumeSpecName "kube-api-access-djnxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:31:25.028585 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.028554    2546 generic.go:356] "Generic (PLEG): container finished" podID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerID="1cfa9775ddd25045538f2c57ab8bfebfc6174acf8af1a40d8f18229dc3090500" exitCode=0
Aug 01 00:31:25.028689 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.028605    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"
Aug 01 00:31:25.028689 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.028629    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" event={"ID":"4abbed71-c3f9-443b-bd60-b1faab1774d7","Type":"ContainerDied","Data":"1cfa9775ddd25045538f2c57ab8bfebfc6174acf8af1a40d8f18229dc3090500"}
Aug 01 00:31:25.028771 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.028695    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5" event={"ID":"4abbed71-c3f9-443b-bd60-b1faab1774d7","Type":"ContainerDied","Data":"8ffa795319c574e1d05614f05d125183e7267006fe48d47faf3a39934587f593"}
Aug 01 00:31:25.028771 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.028733    2546 scope.go:117] "RemoveContainer" containerID="9b85a486266370d6bc065938cbf7f9c1d3841925ff45bc50b48ef577a0ea1a9c"
Aug 01 00:31:25.037623 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.037540    2546 scope.go:117] "RemoveContainer" containerID="1cfa9775ddd25045538f2c57ab8bfebfc6174acf8af1a40d8f18229dc3090500"
Aug 01 00:31:25.044820 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.044803    2546 scope.go:117] "RemoveContainer" containerID="78ab2b3975d04a9a0362329673cd11730e65a812935e14d432d5ffd2771b20ac"
Aug 01 00:31:25.046892 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.046875    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djnxz\" (UniqueName: \"kubernetes.io/projected/4abbed71-c3f9-443b-bd60-b1faab1774d7-kube-api-access-djnxz\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:31:25.046948 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.046895    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4abbed71-c3f9-443b-bd60-b1faab1774d7-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:31:25.046948 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.046904    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4abbed71-c3f9-443b-bd60-b1faab1774d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:31:25.046948 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.046912    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4abbed71-c3f9-443b-bd60-b1faab1774d7-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:31:25.046948 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.046921    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4abbed71-c3f9-443b-bd60-b1faab1774d7-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:31:25.051150 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.051130    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"]
Aug 01 00:31:25.052791 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.052772    2546 scope.go:117] "RemoveContainer" containerID="9b85a486266370d6bc065938cbf7f9c1d3841925ff45bc50b48ef577a0ea1a9c"
Aug 01 00:31:25.053027 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:31:25.053010    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b85a486266370d6bc065938cbf7f9c1d3841925ff45bc50b48ef577a0ea1a9c\": container with ID starting with 9b85a486266370d6bc065938cbf7f9c1d3841925ff45bc50b48ef577a0ea1a9c not found: ID does not exist" containerID="9b85a486266370d6bc065938cbf7f9c1d3841925ff45bc50b48ef577a0ea1a9c"
Aug 01 00:31:25.053103 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.053034    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b85a486266370d6bc065938cbf7f9c1d3841925ff45bc50b48ef577a0ea1a9c"} err="failed to get container status \"9b85a486266370d6bc065938cbf7f9c1d3841925ff45bc50b48ef577a0ea1a9c\": rpc error: code = NotFound desc = could not find container \"9b85a486266370d6bc065938cbf7f9c1d3841925ff45bc50b48ef577a0ea1a9c\": container with ID starting with 9b85a486266370d6bc065938cbf7f9c1d3841925ff45bc50b48ef577a0ea1a9c not found: ID does not exist"
Aug 01 00:31:25.053103 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.053054    2546 scope.go:117] "RemoveContainer" containerID="1cfa9775ddd25045538f2c57ab8bfebfc6174acf8af1a40d8f18229dc3090500"
Aug 01 00:31:25.053320 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:31:25.053294    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfa9775ddd25045538f2c57ab8bfebfc6174acf8af1a40d8f18229dc3090500\": container with ID starting with 1cfa9775ddd25045538f2c57ab8bfebfc6174acf8af1a40d8f18229dc3090500 not found: ID does not exist" containerID="1cfa9775ddd25045538f2c57ab8bfebfc6174acf8af1a40d8f18229dc3090500"
Aug 01 00:31:25.053382 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.053329    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfa9775ddd25045538f2c57ab8bfebfc6174acf8af1a40d8f18229dc3090500"} err="failed to get container status \"1cfa9775ddd25045538f2c57ab8bfebfc6174acf8af1a40d8f18229dc3090500\": rpc error: code = NotFound desc = could not find container \"1cfa9775ddd25045538f2c57ab8bfebfc6174acf8af1a40d8f18229dc3090500\": container with ID starting with 1cfa9775ddd25045538f2c57ab8bfebfc6174acf8af1a40d8f18229dc3090500 not found: ID does not exist"
Aug 01 00:31:25.053382 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.053349    2546 scope.go:117] "RemoveContainer" containerID="78ab2b3975d04a9a0362329673cd11730e65a812935e14d432d5ffd2771b20ac"
Aug 01 00:31:25.053585 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:31:25.053569    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ab2b3975d04a9a0362329673cd11730e65a812935e14d432d5ffd2771b20ac\": container with ID starting with 78ab2b3975d04a9a0362329673cd11730e65a812935e14d432d5ffd2771b20ac not found: ID does not exist" containerID="78ab2b3975d04a9a0362329673cd11730e65a812935e14d432d5ffd2771b20ac"
Aug 01 00:31:25.053634 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.053590    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ab2b3975d04a9a0362329673cd11730e65a812935e14d432d5ffd2771b20ac"} err="failed to get container status \"78ab2b3975d04a9a0362329673cd11730e65a812935e14d432d5ffd2771b20ac\": rpc error: code = NotFound desc = could not find container \"78ab2b3975d04a9a0362329673cd11730e65a812935e14d432d5ffd2771b20ac\": container with ID starting with 78ab2b3975d04a9a0362329673cd11730e65a812935e14d432d5ffd2771b20ac not found: ID does not exist"
Aug 01 00:31:25.054363 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:25.054348    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-6l7x5"]
Aug 01 00:31:26.075372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:26.075341    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" path="/var/lib/kubelet/pods/4abbed71-c3f9-443b-bd60-b1faab1774d7/volumes"
Aug 01 00:31:27.105125 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.105092    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:31:27.129407 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129380    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"]
Aug 01 00:31:27.129673 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129662    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="storage-initializer"
Aug 01 00:31:27.129716 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129675    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="storage-initializer"
Aug 01 00:31:27.129716 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129688    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c48cbe-fa51-4022-ad87-1016d775637b" containerName="kube-rbac-proxy"
Aug 01 00:31:27.129716 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129694    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c48cbe-fa51-4022-ad87-1016d775637b" containerName="kube-rbac-proxy"
Aug 01 00:31:27.129716 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129705    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kserve-container"
Aug 01 00:31:27.129716 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129711    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kserve-container"
Aug 01 00:31:27.129868 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129720    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c48cbe-fa51-4022-ad87-1016d775637b" containerName="storage-initializer"
Aug 01 00:31:27.129868 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129725    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c48cbe-fa51-4022-ad87-1016d775637b" containerName="storage-initializer"
Aug 01 00:31:27.129868 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129734    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c48cbe-fa51-4022-ad87-1016d775637b" containerName="kserve-container"
Aug 01 00:31:27.129868 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129739    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c48cbe-fa51-4022-ad87-1016d775637b" containerName="kserve-container"
Aug 01 00:31:27.129868 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129747    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kube-rbac-proxy"
Aug 01 00:31:27.129868 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129752    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kube-rbac-proxy"
Aug 01 00:31:27.129868 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129797    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kserve-container"
Aug 01 00:31:27.129868 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129803    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c48cbe-fa51-4022-ad87-1016d775637b" containerName="kserve-container"
Aug 01 00:31:27.129868 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129809    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c48cbe-fa51-4022-ad87-1016d775637b" containerName="kube-rbac-proxy"
Aug 01 00:31:27.129868 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.129815    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4abbed71-c3f9-443b-bd60-b1faab1774d7" containerName="kube-rbac-proxy"
Aug 01 00:31:27.140346 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.140315    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:27.141659 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.141637    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"]
Aug 01 00:31:27.142986 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.142964    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:31:27.143113 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.142987    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Aug 01 00:31:27.143113 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.143025    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:31:27.144015 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.143996    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Aug 01 00:31:27.144015 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.144003    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:31:27.144174 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.144080    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Aug 01 00:31:27.263856 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.263836    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8b19f7c-f748-4aa1-b227-386cee5951cc-proxy-tls\") pod \"isvc-sklearn-s3-predictor-54c4564d55-gpgdh\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:27.263956 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.263871    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8b19f7c-f748-4aa1-b227-386cee5951cc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-54c4564d55-gpgdh\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:27.263956 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.263913    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh6rm\" (UniqueName: \"kubernetes.io/projected/b8b19f7c-f748-4aa1-b227-386cee5951cc-kube-api-access-sh6rm\") pod \"isvc-sklearn-s3-predictor-54c4564d55-gpgdh\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:27.264032 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.263949    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8b19f7c-f748-4aa1-b227-386cee5951cc-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-54c4564d55-gpgdh\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:27.364755 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.364737    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh6rm\" (UniqueName: \"kubernetes.io/projected/b8b19f7c-f748-4aa1-b227-386cee5951cc-kube-api-access-sh6rm\") pod \"isvc-sklearn-s3-predictor-54c4564d55-gpgdh\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:27.364842 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.364766    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8b19f7c-f748-4aa1-b227-386cee5951cc-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-54c4564d55-gpgdh\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:27.364842 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.364809    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8b19f7c-f748-4aa1-b227-386cee5951cc-proxy-tls\") pod \"isvc-sklearn-s3-predictor-54c4564d55-gpgdh\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:27.364842 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.364835    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8b19f7c-f748-4aa1-b227-386cee5951cc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-54c4564d55-gpgdh\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:27.364985 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:31:27.364967    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Aug 01 00:31:27.365045 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:31:27.365033    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8b19f7c-f748-4aa1-b227-386cee5951cc-proxy-tls podName:b8b19f7c-f748-4aa1-b227-386cee5951cc nodeName:}" failed. No retries permitted until 2026-08-01 00:31:27.865013509 +0000 UTC m=+2718.324476315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b8b19f7c-f748-4aa1-b227-386cee5951cc-proxy-tls") pod "isvc-sklearn-s3-predictor-54c4564d55-gpgdh" (UID: "b8b19f7c-f748-4aa1-b227-386cee5951cc") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Aug 01 00:31:27.365137 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.365123    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8b19f7c-f748-4aa1-b227-386cee5951cc-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-54c4564d55-gpgdh\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:27.365404 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.365388    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8b19f7c-f748-4aa1-b227-386cee5951cc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-54c4564d55-gpgdh\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:27.372953 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.372938    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh6rm\" (UniqueName: \"kubernetes.io/projected/b8b19f7c-f748-4aa1-b227-386cee5951cc-kube-api-access-sh6rm\") pod \"isvc-sklearn-s3-predictor-54c4564d55-gpgdh\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:27.643289 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.643220    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:31:27.868011 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.867985    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8b19f7c-f748-4aa1-b227-386cee5951cc-proxy-tls\") pod \"isvc-sklearn-s3-predictor-54c4564d55-gpgdh\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:27.870327 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:27.870300    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8b19f7c-f748-4aa1-b227-386cee5951cc-proxy-tls\") pod \"isvc-sklearn-s3-predictor-54c4564d55-gpgdh\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:28.050583 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:28.050518    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:28.176283 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:28.176242    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"]
Aug 01 00:31:28.177470 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:31:28.177445    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b19f7c_f748_4aa1_b227_386cee5951cc.slice/crio-ac16a80b43e041c39fdfba40485d91a90c8cf4a57fbecca533fc5f494902462c WatchSource:0}: Error finding container ac16a80b43e041c39fdfba40485d91a90c8cf4a57fbecca533fc5f494902462c: Status 404 returned error can't find the container with id ac16a80b43e041c39fdfba40485d91a90c8cf4a57fbecca533fc5f494902462c
Aug 01 00:31:28.507418 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:28.507391    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:31:29.046496 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:29.046463    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" event={"ID":"b8b19f7c-f748-4aa1-b227-386cee5951cc","Type":"ContainerStarted","Data":"6981df188949559d47d0ed03e1ecf391b66caed13d3185d3b8ca556ee6d42909"}
Aug 01 00:31:29.046661 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:29.046518    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" event={"ID":"b8b19f7c-f748-4aa1-b227-386cee5951cc","Type":"ContainerStarted","Data":"ac16a80b43e041c39fdfba40485d91a90c8cf4a57fbecca533fc5f494902462c"}
Aug 01 00:31:30.051879 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:30.051849    2546 generic.go:356] "Generic (PLEG): container finished" podID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerID="6981df188949559d47d0ed03e1ecf391b66caed13d3185d3b8ca556ee6d42909" exitCode=0
Aug 01 00:31:30.051879 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:30.051883    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" event={"ID":"b8b19f7c-f748-4aa1-b227-386cee5951cc","Type":"ContainerDied","Data":"6981df188949559d47d0ed03e1ecf391b66caed13d3185d3b8ca556ee6d42909"}
Aug 01 00:31:31.058156 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:31.058123    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" event={"ID":"b8b19f7c-f748-4aa1-b227-386cee5951cc","Type":"ContainerStarted","Data":"3506f90c83fdddbb0b69cb6de1cd3feb07540720bf965333916980726e3aad49"}
Aug 01 00:31:31.058534 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:31.058181    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" event={"ID":"b8b19f7c-f748-4aa1-b227-386cee5951cc","Type":"ContainerStarted","Data":"fdb4b13820a4131bfc7a31fbee8a2cad6795cbc3257a506b3fdc96e5194b56fc"}
Aug 01 00:31:31.058534 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:31.058313    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:31.058534 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:31.058422    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:31.059599 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:31.059576    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Aug 01 00:31:31.076848 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:31.076800    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" podStartSLOduration=3.203031014 podStartE2EDuration="4.076787185s" totalImagesPullingTime="0s" totalInitContainerRuntime="873.756171ms" isStatefulPod=false podCreationTimestamp="2026-08-01 00:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:31:31.075468694 +0000 UTC m=+2721.534931506" watchObservedRunningTime="2026-08-01 00:31:31.076787185 +0000 UTC m=+2721.536249995"
Aug 01 00:31:32.062203 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:32.062166    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Aug 01 00:31:33.631514 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.631485    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:31:33.661812 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.661785    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"]
Aug 01 00:31:33.665366 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.665349    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:33.667512 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.667487    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Aug 01 00:31:33.667644 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.667628    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:31:33.667753 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.667734    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Aug 01 00:31:33.667846 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.667827    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:31:33.676724 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.676705    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"]
Aug 01 00:31:33.709508 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.709484    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:33.709607 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.709512    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww66s\" (UniqueName: \"kubernetes.io/projected/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kube-api-access-ww66s\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:33.709607 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.709534    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:33.709607 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.709550    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:33.709607 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.709605    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:33.810029 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.809996    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww66s\" (UniqueName: \"kubernetes.io/projected/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kube-api-access-ww66s\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:33.810029 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.810037    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:33.810181 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.810054    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:33.810181 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.810094    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:33.810181 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.810173    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:33.810361 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:31:33.810302    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Aug 01 00:31:33.810361 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:31:33.810359    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-proxy-tls podName:d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748 nodeName:}" failed. No retries permitted until 2026-08-01 00:31:34.310341488 +0000 UTC m=+2724.769804292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-cbx77" (UID: "d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748") : secret "isvc-paddle-predictor-serving-cert" not found
Aug 01 00:31:33.810561 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.810543    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:33.810736 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.810721    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:33.810775 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.810720    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:33.820328 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:33.820302    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww66s\" (UniqueName: \"kubernetes.io/projected/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kube-api-access-ww66s\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:34.104816 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:34.104743    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:31:34.312570 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:34.312537    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:34.314912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:34.314888    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-cbx77\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:34.575809 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:34.575778    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:34.698450 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:34.698418    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"]
Aug 01 00:31:34.700221 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:31:34.700194    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a5c312_2a7a_4e6e_9ccf_d41fd02cb748.slice/crio-c7803ac99e2548fc1e78be0e29d40f4e3622c0eec2cde3f77a662e8ec8a5fa9a WatchSource:0}: Error finding container c7803ac99e2548fc1e78be0e29d40f4e3622c0eec2cde3f77a662e8ec8a5fa9a: Status 404 returned error can't find the container with id c7803ac99e2548fc1e78be0e29d40f4e3622c0eec2cde3f77a662e8ec8a5fa9a
Aug 01 00:31:35.074241 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:35.074197    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" event={"ID":"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748","Type":"ContainerStarted","Data":"3aa4f1bbd0155356752e8d87b9de1d8219c80f41a33be1d77f1bc223c901289a"}
Aug 01 00:31:35.074456 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:35.074299    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" event={"ID":"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748","Type":"ContainerStarted","Data":"c7803ac99e2548fc1e78be0e29d40f4e3622c0eec2cde3f77a662e8ec8a5fa9a"}
Aug 01 00:31:37.067897 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:37.067870    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:31:37.068406 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:37.068381    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Aug 01 00:31:40.094146 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:40.094077    2546 generic.go:356] "Generic (PLEG): container finished" podID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerID="3aa4f1bbd0155356752e8d87b9de1d8219c80f41a33be1d77f1bc223c901289a" exitCode=0
Aug 01 00:31:40.094535 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:40.094145    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" event={"ID":"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748","Type":"ContainerDied","Data":"3aa4f1bbd0155356752e8d87b9de1d8219c80f41a33be1d77f1bc223c901289a"}
Aug 01 00:31:40.095197 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:40.095182    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:31:47.069165 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:47.069123    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Aug 01 00:31:52.157441 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:52.157349    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" event={"ID":"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748","Type":"ContainerStarted","Data":"ba35a09f174d038cb9d7bf6ec1bf13dfa5f4497429f3ed7585bb37563259187e"}
Aug 01 00:31:52.157441 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:52.157417    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" event={"ID":"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748","Type":"ContainerStarted","Data":"e9f923c5dc955fd0cc2c8d9a4eddd43b68c3d0b99502ea4b00763f25e7989846"}
Aug 01 00:31:52.157933 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:52.157760    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:52.157933 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:52.157804    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:52.159136 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:52.159112    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Aug 01 00:31:52.177630 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:52.177589    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" podStartSLOduration=2.488050275 podStartE2EDuration="19.177575566s" totalImagesPullingTime="11.781833442s" totalInitContainerRuntime="4.907691849s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:31:52.175831647 +0000 UTC m=+2742.635294459" watchObservedRunningTime="2026-08-01 00:31:52.177575566 +0000 UTC m=+2742.637038376"
Aug 01 00:31:53.163705 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:53.163670    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Aug 01 00:31:57.069159 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:57.069121    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Aug 01 00:31:58.168225 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:58.168200    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:31:58.168770 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:31:58.168746    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Aug 01 00:32:07.069127 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:07.069084    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Aug 01 00:32:08.168650 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:08.168613    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Aug 01 00:32:17.069275 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:17.069220    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Aug 01 00:32:18.169035 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:18.168998    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Aug 01 00:32:27.068456 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:27.068416    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Aug 01 00:32:28.169507 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:28.169469    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Aug 01 00:32:37.068659 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:37.068616    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Aug 01 00:32:38.169405 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:38.169377    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:32:45.359822 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:45.359782    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"]
Aug 01 00:32:45.360333 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:45.360094    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="kserve-container" containerID="cri-o://e9f923c5dc955fd0cc2c8d9a4eddd43b68c3d0b99502ea4b00763f25e7989846" gracePeriod=30
Aug 01 00:32:45.360333 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:45.360119    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="kube-rbac-proxy" containerID="cri-o://ba35a09f174d038cb9d7bf6ec1bf13dfa5f4497429f3ed7585bb37563259187e" gracePeriod=30
Aug 01 00:32:45.381030 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:45.380998    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:46.384775 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:46.384744    2546 generic.go:356] "Generic (PLEG): container finished" podID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerID="ba35a09f174d038cb9d7bf6ec1bf13dfa5f4497429f3ed7585bb37563259187e" exitCode=2
Aug 01 00:32:46.385176 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:46.384815    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" event={"ID":"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748","Type":"ContainerDied","Data":"ba35a09f174d038cb9d7bf6ec1bf13dfa5f4497429f3ed7585bb37563259187e"}
Aug 01 00:32:47.068948 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:47.068914    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:32:48.100419 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.100395    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:32:48.116630 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.116597    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-cabundle-cert\") pod \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") "
Aug 01 00:32:48.116630 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.116634    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kube-rbac-proxy-sar-config\") pod \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") "
Aug 01 00:32:48.116867 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.116678    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-proxy-tls\") pod \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") "
Aug 01 00:32:48.116867 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.116695    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww66s\" (UniqueName: \"kubernetes.io/projected/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kube-api-access-ww66s\") pod \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") "
Aug 01 00:32:48.116867 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.116733    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kserve-provision-location\") pod \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\" (UID: \"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748\") "
Aug 01 00:32:48.117118 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.117053    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" (UID: "d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:32:48.117118 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.117098    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" (UID: "d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:32:48.119495 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.119468    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kube-api-access-ww66s" (OuterVolumeSpecName: "kube-api-access-ww66s") pod "d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" (UID: "d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748"). InnerVolumeSpecName "kube-api-access-ww66s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:32:48.119997 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.119970    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" (UID: "d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:32:48.127904 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.127867    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" (UID: "d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:32:48.217635 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.217552    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:32:48.217635 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.217580    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:32:48.217635 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.217589    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww66s\" (UniqueName: \"kubernetes.io/projected/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kube-api-access-ww66s\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:32:48.217635 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.217598    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:32:48.217635 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.217606    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:32:48.397399 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.396532    2546 generic.go:356] "Generic (PLEG): container finished" podID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerID="e9f923c5dc955fd0cc2c8d9a4eddd43b68c3d0b99502ea4b00763f25e7989846" exitCode=0
Aug 01 00:32:48.397399 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.396798    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" event={"ID":"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748","Type":"ContainerDied","Data":"e9f923c5dc955fd0cc2c8d9a4eddd43b68c3d0b99502ea4b00763f25e7989846"}
Aug 01 00:32:48.397399 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.396927    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77" event={"ID":"d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748","Type":"ContainerDied","Data":"c7803ac99e2548fc1e78be0e29d40f4e3622c0eec2cde3f77a662e8ec8a5fa9a"}
Aug 01 00:32:48.397399 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.396982    2546 scope.go:117] "RemoveContainer" containerID="ba35a09f174d038cb9d7bf6ec1bf13dfa5f4497429f3ed7585bb37563259187e"
Aug 01 00:32:48.397399 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.396985    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"
Aug 01 00:32:48.408477 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.408446    2546 scope.go:117] "RemoveContainer" containerID="e9f923c5dc955fd0cc2c8d9a4eddd43b68c3d0b99502ea4b00763f25e7989846"
Aug 01 00:32:48.416982 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.416962    2546 scope.go:117] "RemoveContainer" containerID="3aa4f1bbd0155356752e8d87b9de1d8219c80f41a33be1d77f1bc223c901289a"
Aug 01 00:32:48.424306 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.424280    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"]
Aug 01 00:32:48.425832 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.425618    2546 scope.go:117] "RemoveContainer" containerID="ba35a09f174d038cb9d7bf6ec1bf13dfa5f4497429f3ed7585bb37563259187e"
Aug 01 00:32:48.426154 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:32:48.426075    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba35a09f174d038cb9d7bf6ec1bf13dfa5f4497429f3ed7585bb37563259187e\": container with ID starting with ba35a09f174d038cb9d7bf6ec1bf13dfa5f4497429f3ed7585bb37563259187e not found: ID does not exist" containerID="ba35a09f174d038cb9d7bf6ec1bf13dfa5f4497429f3ed7585bb37563259187e"
Aug 01 00:32:48.426240 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.426166    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba35a09f174d038cb9d7bf6ec1bf13dfa5f4497429f3ed7585bb37563259187e"} err="failed to get container status \"ba35a09f174d038cb9d7bf6ec1bf13dfa5f4497429f3ed7585bb37563259187e\": rpc error: code = NotFound desc = could not find container \"ba35a09f174d038cb9d7bf6ec1bf13dfa5f4497429f3ed7585bb37563259187e\": container with ID starting with ba35a09f174d038cb9d7bf6ec1bf13dfa5f4497429f3ed7585bb37563259187e not found: ID does not exist"
Aug 01 00:32:48.426240 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.426198    2546 scope.go:117] "RemoveContainer" containerID="e9f923c5dc955fd0cc2c8d9a4eddd43b68c3d0b99502ea4b00763f25e7989846"
Aug 01 00:32:48.426495 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:32:48.426480    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f923c5dc955fd0cc2c8d9a4eddd43b68c3d0b99502ea4b00763f25e7989846\": container with ID starting with e9f923c5dc955fd0cc2c8d9a4eddd43b68c3d0b99502ea4b00763f25e7989846 not found: ID does not exist" containerID="e9f923c5dc955fd0cc2c8d9a4eddd43b68c3d0b99502ea4b00763f25e7989846"
Aug 01 00:32:48.426540 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.426502    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f923c5dc955fd0cc2c8d9a4eddd43b68c3d0b99502ea4b00763f25e7989846"} err="failed to get container status \"e9f923c5dc955fd0cc2c8d9a4eddd43b68c3d0b99502ea4b00763f25e7989846\": rpc error: code = NotFound desc = could not find container \"e9f923c5dc955fd0cc2c8d9a4eddd43b68c3d0b99502ea4b00763f25e7989846\": container with ID starting with e9f923c5dc955fd0cc2c8d9a4eddd43b68c3d0b99502ea4b00763f25e7989846 not found: ID does not exist"
Aug 01 00:32:48.426540 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.426518    2546 scope.go:117] "RemoveContainer" containerID="3aa4f1bbd0155356752e8d87b9de1d8219c80f41a33be1d77f1bc223c901289a"
Aug 01 00:32:48.426769 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:32:48.426749    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa4f1bbd0155356752e8d87b9de1d8219c80f41a33be1d77f1bc223c901289a\": container with ID starting with 3aa4f1bbd0155356752e8d87b9de1d8219c80f41a33be1d77f1bc223c901289a not found: ID does not exist" containerID="3aa4f1bbd0155356752e8d87b9de1d8219c80f41a33be1d77f1bc223c901289a"
Aug 01 00:32:48.426821 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.426776    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa4f1bbd0155356752e8d87b9de1d8219c80f41a33be1d77f1bc223c901289a"} err="failed to get container status \"3aa4f1bbd0155356752e8d87b9de1d8219c80f41a33be1d77f1bc223c901289a\": rpc error: code = NotFound desc = could not find container \"3aa4f1bbd0155356752e8d87b9de1d8219c80f41a33be1d77f1bc223c901289a\": container with ID starting with 3aa4f1bbd0155356752e8d87b9de1d8219c80f41a33be1d77f1bc223c901289a not found: ID does not exist"
Aug 01 00:32:48.427678 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:48.427660    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-cbx77"]
Aug 01 00:32:50.075812 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:50.075779    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" path="/var/lib/kubelet/pods/d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748/volumes"
Aug 01 00:32:50.501589 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:50.501528    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:50.645485 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:50.645455    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:57.238107 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.238077    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"]
Aug 01 00:32:57.238597 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.238398    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kserve-container" containerID="cri-o://fdb4b13820a4131bfc7a31fbee8a2cad6795cbc3257a506b3fdc96e5194b56fc" gracePeriod=30
Aug 01 00:32:57.238597 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.238419    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kube-rbac-proxy" containerID="cri-o://3506f90c83fdddbb0b69cb6de1cd3feb07540720bf965333916980726e3aad49" gracePeriod=30
Aug 01 00:32:57.251703 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.251680    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:57.377202 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.377166    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:57.399940 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.399907    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"]
Aug 01 00:32:57.400312 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.400297    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="storage-initializer"
Aug 01 00:32:57.400403 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.400314    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="storage-initializer"
Aug 01 00:32:57.400403 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.400329    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="kube-rbac-proxy"
Aug 01 00:32:57.400403 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.400338    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="kube-rbac-proxy"
Aug 01 00:32:57.400403 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.400374    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="kserve-container"
Aug 01 00:32:57.400403 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.400383    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="kserve-container"
Aug 01 00:32:57.400672 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.400470    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="kserve-container"
Aug 01 00:32:57.400672 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.400482    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2a5c312-2a7a-4e6e-9ccf-d41fd02cb748" containerName="kube-rbac-proxy"
Aug 01 00:32:57.402537 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.402520    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.404878 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.404858    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Aug 01 00:32:57.405076 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.405059    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:32:57.405156 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.405104    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Aug 01 00:32:57.411956 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.411934    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"]
Aug 01 00:32:57.437532 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.437505    2546 generic.go:356] "Generic (PLEG): container finished" podID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerID="3506f90c83fdddbb0b69cb6de1cd3feb07540720bf965333916980726e3aad49" exitCode=2
Aug 01 00:32:57.437675 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.437565    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" event={"ID":"b8b19f7c-f748-4aa1-b227-386cee5951cc","Type":"ContainerDied","Data":"3506f90c83fdddbb0b69cb6de1cd3feb07540720bf965333916980726e3aad49"}
Aug 01 00:32:57.498010 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.497933    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8969d00d-dfae-4aae-ab0f-31f4375a6afa-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.498010 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.497965    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.498010 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.497996    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9czt\" (UniqueName: \"kubernetes.io/projected/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kube-api-access-n9czt\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.498206 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.498051    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.498206 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.498137    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8969d00d-dfae-4aae-ab0f-31f4375a6afa-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.567540 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.567511    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:32:57.598782 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.598758    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8969d00d-dfae-4aae-ab0f-31f4375a6afa-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.598890 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.598806    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8969d00d-dfae-4aae-ab0f-31f4375a6afa-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.598890 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.598824    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.598890 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.598842    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9czt\" (UniqueName: \"kubernetes.io/projected/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kube-api-access-n9czt\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.598890 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.598869    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.599327 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.599307    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.599631 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.599609    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8969d00d-dfae-4aae-ab0f-31f4375a6afa-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.599679 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.599609    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.601315 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.601293    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8969d00d-dfae-4aae-ab0f-31f4375a6afa-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.606182 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.606162    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9czt\" (UniqueName: \"kubernetes.io/projected/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kube-api-access-n9czt\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.712890 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.712855    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:32:57.839284 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:57.839247    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"]
Aug 01 00:32:57.841384 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:32:57.841359    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8969d00d_dfae_4aae_ab0f_31f4375a6afa.slice/crio-cecb0be968d11f3e24c61d07048196736f9742dd3ce3d4c1e29a94113ee025e7 WatchSource:0}: Error finding container cecb0be968d11f3e24c61d07048196736f9742dd3ce3d4c1e29a94113ee025e7: Status 404 returned error can't find the container with id cecb0be968d11f3e24c61d07048196736f9742dd3ce3d4c1e29a94113ee025e7
Aug 01 00:32:58.444000 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:58.443963    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" event={"ID":"8969d00d-dfae-4aae-ab0f-31f4375a6afa","Type":"ContainerStarted","Data":"b5b7b670b55bc5647dd3d2a79405e7ca6f52ce8c1ddf8101bda7d49e3e8adb9a"}
Aug 01 00:32:58.444461 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:58.444024    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" event={"ID":"8969d00d-dfae-4aae-ab0f-31f4375a6afa","Type":"ContainerStarted","Data":"cecb0be968d11f3e24c61d07048196736f9742dd3ce3d4c1e29a94113ee025e7"}
Aug 01 00:32:59.449492 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:59.449401    2546 generic.go:356] "Generic (PLEG): container finished" podID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerID="b5b7b670b55bc5647dd3d2a79405e7ca6f52ce8c1ddf8101bda7d49e3e8adb9a" exitCode=0
Aug 01 00:32:59.449848 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:32:59.449489    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" event={"ID":"8969d00d-dfae-4aae-ab0f-31f4375a6afa","Type":"ContainerDied","Data":"b5b7b670b55bc5647dd3d2a79405e7ca6f52ce8c1ddf8101bda7d49e3e8adb9a"}
Aug 01 00:33:00.455919 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:00.455885    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" event={"ID":"8969d00d-dfae-4aae-ab0f-31f4375a6afa","Type":"ContainerStarted","Data":"c26e09fdaccd9c41f55a51ffcf72094aac7368468a8e61f29ed946cc7225a80d"}
Aug 01 00:33:00.456398 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:00.455944    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" event={"ID":"8969d00d-dfae-4aae-ab0f-31f4375a6afa","Type":"ContainerStarted","Data":"c1f9602033aaa166c0c32a4d95393a0c597768211442e8be1942dbe9d7828f06"}
Aug 01 00:33:00.456398 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:00.455971    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:33:00.456398 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:00.456126    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:33:00.457691 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:00.457663    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Aug 01 00:33:00.476926 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:00.476867    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" podStartSLOduration=2.5158347279999997 podStartE2EDuration="3.476853463s" totalImagesPullingTime="0s" totalInitContainerRuntime="961.018735ms" isStatefulPod=false podCreationTimestamp="2026-08-01 00:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:33:00.475432422 +0000 UTC m=+2810.934895234" watchObservedRunningTime="2026-08-01 00:33:00.476853463 +0000 UTC m=+2810.936316275"
Aug 01 00:33:01.459816 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:01.459760    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Aug 01 00:33:01.984764 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:01.984741    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:33:02.039434 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.039404    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8b19f7c-f748-4aa1-b227-386cee5951cc-kserve-provision-location\") pod \"b8b19f7c-f748-4aa1-b227-386cee5951cc\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") "
Aug 01 00:33:02.039624 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.039446    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh6rm\" (UniqueName: \"kubernetes.io/projected/b8b19f7c-f748-4aa1-b227-386cee5951cc-kube-api-access-sh6rm\") pod \"b8b19f7c-f748-4aa1-b227-386cee5951cc\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") "
Aug 01 00:33:02.039624 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.039476    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8b19f7c-f748-4aa1-b227-386cee5951cc-kube-rbac-proxy-sar-config\") pod \"b8b19f7c-f748-4aa1-b227-386cee5951cc\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") "
Aug 01 00:33:02.039624 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.039498    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8b19f7c-f748-4aa1-b227-386cee5951cc-proxy-tls\") pod \"b8b19f7c-f748-4aa1-b227-386cee5951cc\" (UID: \"b8b19f7c-f748-4aa1-b227-386cee5951cc\") "
Aug 01 00:33:02.039804 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.039759    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b19f7c-f748-4aa1-b227-386cee5951cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8b19f7c-f748-4aa1-b227-386cee5951cc" (UID: "b8b19f7c-f748-4aa1-b227-386cee5951cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:33:02.039851 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.039822    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b19f7c-f748-4aa1-b227-386cee5951cc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8b19f7c-f748-4aa1-b227-386cee5951cc" (UID: "b8b19f7c-f748-4aa1-b227-386cee5951cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:33:02.041591 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.041571    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b19f7c-f748-4aa1-b227-386cee5951cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8b19f7c-f748-4aa1-b227-386cee5951cc" (UID: "b8b19f7c-f748-4aa1-b227-386cee5951cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:33:02.041699 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.041624    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b19f7c-f748-4aa1-b227-386cee5951cc-kube-api-access-sh6rm" (OuterVolumeSpecName: "kube-api-access-sh6rm") pod "b8b19f7c-f748-4aa1-b227-386cee5951cc" (UID: "b8b19f7c-f748-4aa1-b227-386cee5951cc"). InnerVolumeSpecName "kube-api-access-sh6rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:33:02.140356 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.140319    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8b19f7c-f748-4aa1-b227-386cee5951cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:33:02.140356 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.140351    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8b19f7c-f748-4aa1-b227-386cee5951cc-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:33:02.140551 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.140367    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8b19f7c-f748-4aa1-b227-386cee5951cc-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:33:02.140551 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.140379    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh6rm\" (UniqueName: \"kubernetes.io/projected/b8b19f7c-f748-4aa1-b227-386cee5951cc-kube-api-access-sh6rm\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:33:02.465390 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.465350    2546 generic.go:356] "Generic (PLEG): container finished" podID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerID="fdb4b13820a4131bfc7a31fbee8a2cad6795cbc3257a506b3fdc96e5194b56fc" exitCode=0
Aug 01 00:33:02.465829 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.465415    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"
Aug 01 00:33:02.465829 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.465438    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" event={"ID":"b8b19f7c-f748-4aa1-b227-386cee5951cc","Type":"ContainerDied","Data":"fdb4b13820a4131bfc7a31fbee8a2cad6795cbc3257a506b3fdc96e5194b56fc"}
Aug 01 00:33:02.465829 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.465520    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh" event={"ID":"b8b19f7c-f748-4aa1-b227-386cee5951cc","Type":"ContainerDied","Data":"ac16a80b43e041c39fdfba40485d91a90c8cf4a57fbecca533fc5f494902462c"}
Aug 01 00:33:02.465829 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.465562    2546 scope.go:117] "RemoveContainer" containerID="3506f90c83fdddbb0b69cb6de1cd3feb07540720bf965333916980726e3aad49"
Aug 01 00:33:02.473568 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.473551    2546 scope.go:117] "RemoveContainer" containerID="fdb4b13820a4131bfc7a31fbee8a2cad6795cbc3257a506b3fdc96e5194b56fc"
Aug 01 00:33:02.481226 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.481206    2546 scope.go:117] "RemoveContainer" containerID="6981df188949559d47d0ed03e1ecf391b66caed13d3185d3b8ca556ee6d42909"
Aug 01 00:33:02.482662 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.482631    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"]
Aug 01 00:33:02.486040 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.486019    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-54c4564d55-gpgdh"]
Aug 01 00:33:02.488633 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.488619    2546 scope.go:117] "RemoveContainer" containerID="3506f90c83fdddbb0b69cb6de1cd3feb07540720bf965333916980726e3aad49"
Aug 01 00:33:02.488947 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:33:02.488928    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3506f90c83fdddbb0b69cb6de1cd3feb07540720bf965333916980726e3aad49\": container with ID starting with 3506f90c83fdddbb0b69cb6de1cd3feb07540720bf965333916980726e3aad49 not found: ID does not exist" containerID="3506f90c83fdddbb0b69cb6de1cd3feb07540720bf965333916980726e3aad49"
Aug 01 00:33:02.489000 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.488956    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3506f90c83fdddbb0b69cb6de1cd3feb07540720bf965333916980726e3aad49"} err="failed to get container status \"3506f90c83fdddbb0b69cb6de1cd3feb07540720bf965333916980726e3aad49\": rpc error: code = NotFound desc = could not find container \"3506f90c83fdddbb0b69cb6de1cd3feb07540720bf965333916980726e3aad49\": container with ID starting with 3506f90c83fdddbb0b69cb6de1cd3feb07540720bf965333916980726e3aad49 not found: ID does not exist"
Aug 01 00:33:02.489000 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.488973    2546 scope.go:117] "RemoveContainer" containerID="fdb4b13820a4131bfc7a31fbee8a2cad6795cbc3257a506b3fdc96e5194b56fc"
Aug 01 00:33:02.489217 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:33:02.489198    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb4b13820a4131bfc7a31fbee8a2cad6795cbc3257a506b3fdc96e5194b56fc\": container with ID starting with fdb4b13820a4131bfc7a31fbee8a2cad6795cbc3257a506b3fdc96e5194b56fc not found: ID does not exist" containerID="fdb4b13820a4131bfc7a31fbee8a2cad6795cbc3257a506b3fdc96e5194b56fc"
Aug 01 00:33:02.489281 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.489223    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb4b13820a4131bfc7a31fbee8a2cad6795cbc3257a506b3fdc96e5194b56fc"} err="failed to get container status \"fdb4b13820a4131bfc7a31fbee8a2cad6795cbc3257a506b3fdc96e5194b56fc\": rpc error: code = NotFound desc = could not find container \"fdb4b13820a4131bfc7a31fbee8a2cad6795cbc3257a506b3fdc96e5194b56fc\": container with ID starting with fdb4b13820a4131bfc7a31fbee8a2cad6795cbc3257a506b3fdc96e5194b56fc not found: ID does not exist"
Aug 01 00:33:02.489281 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.489239    2546 scope.go:117] "RemoveContainer" containerID="6981df188949559d47d0ed03e1ecf391b66caed13d3185d3b8ca556ee6d42909"
Aug 01 00:33:02.489489 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:33:02.489472    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6981df188949559d47d0ed03e1ecf391b66caed13d3185d3b8ca556ee6d42909\": container with ID starting with 6981df188949559d47d0ed03e1ecf391b66caed13d3185d3b8ca556ee6d42909 not found: ID does not exist" containerID="6981df188949559d47d0ed03e1ecf391b66caed13d3185d3b8ca556ee6d42909"
Aug 01 00:33:02.489535 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:02.489493    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6981df188949559d47d0ed03e1ecf391b66caed13d3185d3b8ca556ee6d42909"} err="failed to get container status \"6981df188949559d47d0ed03e1ecf391b66caed13d3185d3b8ca556ee6d42909\": rpc error: code = NotFound desc = could not find container \"6981df188949559d47d0ed03e1ecf391b66caed13d3185d3b8ca556ee6d42909\": container with ID starting with 6981df188949559d47d0ed03e1ecf391b66caed13d3185d3b8ca556ee6d42909 not found: ID does not exist"
Aug 01 00:33:04.075631 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:04.075597    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" path="/var/lib/kubelet/pods/b8b19f7c-f748-4aa1-b227-386cee5951cc/volumes"
Aug 01 00:33:06.464023 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:06.463995    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:33:06.464615 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:06.464587    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Aug 01 00:33:16.464974 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:16.464934    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Aug 01 00:33:26.465452 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:26.465407    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Aug 01 00:33:36.464948 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:36.464906    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Aug 01 00:33:46.464717 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:46.464676    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Aug 01 00:33:56.465379 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:33:56.465335    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Aug 01 00:34:02.269267 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:02.269218    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:06.464925 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:06.464878    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Aug 01 00:34:07.466804 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:07.466774    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:07.646904 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:07.646871    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:16.465141 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:16.465113    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:34:17.425201 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:17.425167    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"]
Aug 01 00:34:17.425502 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:17.425468    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kserve-container" containerID="cri-o://c1f9602033aaa166c0c32a4d95393a0c597768211442e8be1942dbe9d7828f06" gracePeriod=30
Aug 01 00:34:17.425631 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:17.425520    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kube-rbac-proxy" containerID="cri-o://c26e09fdaccd9c41f55a51ffcf72094aac7368468a8e61f29ed946cc7225a80d" gracePeriod=30
Aug 01 00:34:17.453909 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:17.453871    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:17.777535 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:17.777462    2546 generic.go:356] "Generic (PLEG): container finished" podID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerID="c26e09fdaccd9c41f55a51ffcf72094aac7368468a8e61f29ed946cc7225a80d" exitCode=2
Aug 01 00:34:17.777851 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:17.777534    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" event={"ID":"8969d00d-dfae-4aae-ab0f-31f4375a6afa","Type":"ContainerDied","Data":"c26e09fdaccd9c41f55a51ffcf72094aac7368468a8e61f29ed946cc7225a80d"}
Aug 01 00:34:18.501578 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.501544    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"]
Aug 01 00:34:18.501875 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.501863    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kube-rbac-proxy"
Aug 01 00:34:18.501928 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.501877    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kube-rbac-proxy"
Aug 01 00:34:18.501928 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.501890    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="storage-initializer"
Aug 01 00:34:18.501928 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.501896    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="storage-initializer"
Aug 01 00:34:18.501928 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.501913    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kserve-container"
Aug 01 00:34:18.501928 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.501918    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kserve-container"
Aug 01 00:34:18.502084 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.501968    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kserve-container"
Aug 01 00:34:18.502084 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.501976    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b19f7c-f748-4aa1-b227-386cee5951cc" containerName="kube-rbac-proxy"
Aug 01 00:34:18.504933 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.504918    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:18.507431 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.507410    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Aug 01 00:34:18.507555 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.507507    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Aug 01 00:34:18.516383 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.516365    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"]
Aug 01 00:34:18.629275 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.629235    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c965756-5169-45b8-9415-7d226a624cc0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:18.629395 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.629310    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkmrw\" (UniqueName: \"kubernetes.io/projected/5c965756-5169-45b8-9415-7d226a624cc0-kube-api-access-pkmrw\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:18.629437 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.629408    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c965756-5169-45b8-9415-7d226a624cc0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:18.629482 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.629440    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c965756-5169-45b8-9415-7d226a624cc0-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:18.730069 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.730044    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c965756-5169-45b8-9415-7d226a624cc0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:18.730206 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.730094    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c965756-5169-45b8-9415-7d226a624cc0-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:18.730206 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.730154    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c965756-5169-45b8-9415-7d226a624cc0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:18.730206 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.730185    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkmrw\" (UniqueName: \"kubernetes.io/projected/5c965756-5169-45b8-9415-7d226a624cc0-kube-api-access-pkmrw\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:18.730392 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:34:18.730291    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Aug 01 00:34:18.730392 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:34:18.730371    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c965756-5169-45b8-9415-7d226a624cc0-proxy-tls podName:5c965756-5169-45b8-9415-7d226a624cc0 nodeName:}" failed. No retries permitted until 2026-08-01 00:34:19.230343423 +0000 UTC m=+2889.689806214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5c965756-5169-45b8-9415-7d226a624cc0-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm" (UID: "5c965756-5169-45b8-9415-7d226a624cc0") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Aug 01 00:34:18.730502 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.730465    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c965756-5169-45b8-9415-7d226a624cc0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:18.730780 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.730763    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c965756-5169-45b8-9415-7d226a624cc0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:18.740171 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:18.740151    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkmrw\" (UniqueName: \"kubernetes.io/projected/5c965756-5169-45b8-9415-7d226a624cc0-kube-api-access-pkmrw\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:19.117837 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:19.117808    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:19.234554 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:19.234525    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c965756-5169-45b8-9415-7d226a624cc0-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:19.236929 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:19.236904    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c965756-5169-45b8-9415-7d226a624cc0-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:19.415422 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:19.415347    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:19.539412 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:19.539389    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"]
Aug 01 00:34:19.541218 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:34:19.541188    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c965756_5169_45b8_9415_7d226a624cc0.slice/crio-43cfcb885b5337618481aa7dd56b6c078c9653721272b5826c14778b1547d277 WatchSource:0}: Error finding container 43cfcb885b5337618481aa7dd56b6c078c9653721272b5826c14778b1547d277: Status 404 returned error can't find the container with id 43cfcb885b5337618481aa7dd56b6c078c9653721272b5826c14778b1547d277
Aug 01 00:34:19.787315 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:19.787224    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm" event={"ID":"5c965756-5169-45b8-9415-7d226a624cc0","Type":"ContainerStarted","Data":"54dab054c5f4c24499b71e31a57b8d6677f03616dd5f5e77e7bd540e8900519f"}
Aug 01 00:34:19.787315 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:19.787303    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm" event={"ID":"5c965756-5169-45b8-9415-7d226a624cc0","Type":"ContainerStarted","Data":"43cfcb885b5337618481aa7dd56b6c078c9653721272b5826c14778b1547d277"}
Aug 01 00:34:21.460436 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.460397    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.63:8643/healthz\": dial tcp 10.133.0.63:8643: connect: connection refused"
Aug 01 00:34:21.575564 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.575541    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:34:21.652171 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.652150    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kube-rbac-proxy-sar-config\") pod \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") "
Aug 01 00:34:21.652318 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.652176    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8969d00d-dfae-4aae-ab0f-31f4375a6afa-cabundle-cert\") pod \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") "
Aug 01 00:34:21.652318 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.652197    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8969d00d-dfae-4aae-ab0f-31f4375a6afa-proxy-tls\") pod \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") "
Aug 01 00:34:21.652318 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.652213    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kserve-provision-location\") pod \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") "
Aug 01 00:34:21.652318 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.652246    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9czt\" (UniqueName: \"kubernetes.io/projected/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kube-api-access-n9czt\") pod \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\" (UID: \"8969d00d-dfae-4aae-ab0f-31f4375a6afa\") "
Aug 01 00:34:21.652604 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.652481    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8969d00d-dfae-4aae-ab0f-31f4375a6afa" (UID: "8969d00d-dfae-4aae-ab0f-31f4375a6afa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:34:21.652657 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.652634    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8969d00d-dfae-4aae-ab0f-31f4375a6afa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8969d00d-dfae-4aae-ab0f-31f4375a6afa" (UID: "8969d00d-dfae-4aae-ab0f-31f4375a6afa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:34:21.652696 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.652673    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8969d00d-dfae-4aae-ab0f-31f4375a6afa" (UID: "8969d00d-dfae-4aae-ab0f-31f4375a6afa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:34:21.654203 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.654179    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8969d00d-dfae-4aae-ab0f-31f4375a6afa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8969d00d-dfae-4aae-ab0f-31f4375a6afa" (UID: "8969d00d-dfae-4aae-ab0f-31f4375a6afa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:34:21.654281 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.654228    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kube-api-access-n9czt" (OuterVolumeSpecName: "kube-api-access-n9czt") pod "8969d00d-dfae-4aae-ab0f-31f4375a6afa" (UID: "8969d00d-dfae-4aae-ab0f-31f4375a6afa"). InnerVolumeSpecName "kube-api-access-n9czt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:34:21.753310 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.753247    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:34:21.753310 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.753282    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8969d00d-dfae-4aae-ab0f-31f4375a6afa-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:34:21.753310 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.753292    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8969d00d-dfae-4aae-ab0f-31f4375a6afa-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:34:21.753310 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.753299    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:34:21.753310 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.753307    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9czt\" (UniqueName: \"kubernetes.io/projected/8969d00d-dfae-4aae-ab0f-31f4375a6afa-kube-api-access-n9czt\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:34:21.796237 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.796210    2546 generic.go:356] "Generic (PLEG): container finished" podID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerID="c1f9602033aaa166c0c32a4d95393a0c597768211442e8be1942dbe9d7828f06" exitCode=0
Aug 01 00:34:21.796329 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.796272    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"
Aug 01 00:34:21.796329 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.796294    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" event={"ID":"8969d00d-dfae-4aae-ab0f-31f4375a6afa","Type":"ContainerDied","Data":"c1f9602033aaa166c0c32a4d95393a0c597768211442e8be1942dbe9d7828f06"}
Aug 01 00:34:21.796407 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.796393    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv" event={"ID":"8969d00d-dfae-4aae-ab0f-31f4375a6afa","Type":"ContainerDied","Data":"cecb0be968d11f3e24c61d07048196736f9742dd3ce3d4c1e29a94113ee025e7"}
Aug 01 00:34:21.796454 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.796434    2546 scope.go:117] "RemoveContainer" containerID="c26e09fdaccd9c41f55a51ffcf72094aac7368468a8e61f29ed946cc7225a80d"
Aug 01 00:34:21.804442 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.804427    2546 scope.go:117] "RemoveContainer" containerID="c1f9602033aaa166c0c32a4d95393a0c597768211442e8be1942dbe9d7828f06"
Aug 01 00:34:21.811551 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.811538    2546 scope.go:117] "RemoveContainer" containerID="b5b7b670b55bc5647dd3d2a79405e7ca6f52ce8c1ddf8101bda7d49e3e8adb9a"
Aug 01 00:34:21.818206 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.818189    2546 scope.go:117] "RemoveContainer" containerID="c26e09fdaccd9c41f55a51ffcf72094aac7368468a8e61f29ed946cc7225a80d"
Aug 01 00:34:21.818448 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:34:21.818429    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c26e09fdaccd9c41f55a51ffcf72094aac7368468a8e61f29ed946cc7225a80d\": container with ID starting with c26e09fdaccd9c41f55a51ffcf72094aac7368468a8e61f29ed946cc7225a80d not found: ID does not exist" containerID="c26e09fdaccd9c41f55a51ffcf72094aac7368468a8e61f29ed946cc7225a80d"
Aug 01 00:34:21.818503 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.818455    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c26e09fdaccd9c41f55a51ffcf72094aac7368468a8e61f29ed946cc7225a80d"} err="failed to get container status \"c26e09fdaccd9c41f55a51ffcf72094aac7368468a8e61f29ed946cc7225a80d\": rpc error: code = NotFound desc = could not find container \"c26e09fdaccd9c41f55a51ffcf72094aac7368468a8e61f29ed946cc7225a80d\": container with ID starting with c26e09fdaccd9c41f55a51ffcf72094aac7368468a8e61f29ed946cc7225a80d not found: ID does not exist"
Aug 01 00:34:21.818503 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.818471    2546 scope.go:117] "RemoveContainer" containerID="c1f9602033aaa166c0c32a4d95393a0c597768211442e8be1942dbe9d7828f06"
Aug 01 00:34:21.818695 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:34:21.818679    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f9602033aaa166c0c32a4d95393a0c597768211442e8be1942dbe9d7828f06\": container with ID starting with c1f9602033aaa166c0c32a4d95393a0c597768211442e8be1942dbe9d7828f06 not found: ID does not exist" containerID="c1f9602033aaa166c0c32a4d95393a0c597768211442e8be1942dbe9d7828f06"
Aug 01 00:34:21.818740 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.818701    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f9602033aaa166c0c32a4d95393a0c597768211442e8be1942dbe9d7828f06"} err="failed to get container status \"c1f9602033aaa166c0c32a4d95393a0c597768211442e8be1942dbe9d7828f06\": rpc error: code = NotFound desc = could not find container \"c1f9602033aaa166c0c32a4d95393a0c597768211442e8be1942dbe9d7828f06\": container with ID starting with c1f9602033aaa166c0c32a4d95393a0c597768211442e8be1942dbe9d7828f06 not found: ID does not exist"
Aug 01 00:34:21.818740 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.818715    2546 scope.go:117] "RemoveContainer" containerID="b5b7b670b55bc5647dd3d2a79405e7ca6f52ce8c1ddf8101bda7d49e3e8adb9a"
Aug 01 00:34:21.818941 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:34:21.818921    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b7b670b55bc5647dd3d2a79405e7ca6f52ce8c1ddf8101bda7d49e3e8adb9a\": container with ID starting with b5b7b670b55bc5647dd3d2a79405e7ca6f52ce8c1ddf8101bda7d49e3e8adb9a not found: ID does not exist" containerID="b5b7b670b55bc5647dd3d2a79405e7ca6f52ce8c1ddf8101bda7d49e3e8adb9a"
Aug 01 00:34:21.818988 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.818949    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b7b670b55bc5647dd3d2a79405e7ca6f52ce8c1ddf8101bda7d49e3e8adb9a"} err="failed to get container status \"b5b7b670b55bc5647dd3d2a79405e7ca6f52ce8c1ddf8101bda7d49e3e8adb9a\": rpc error: code = NotFound desc = could not find container \"b5b7b670b55bc5647dd3d2a79405e7ca6f52ce8c1ddf8101bda7d49e3e8adb9a\": container with ID starting with b5b7b670b55bc5647dd3d2a79405e7ca6f52ce8c1ddf8101bda7d49e3e8adb9a not found: ID does not exist"
Aug 01 00:34:21.827711 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.827689    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"]
Aug 01 00:34:21.837634 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:21.837615    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-95f84c47f-vwggv"]
Aug 01 00:34:22.075841 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:22.075773    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" path="/var/lib/kubelet/pods/8969d00d-dfae-4aae-ab0f-31f4375a6afa/volumes"
Aug 01 00:34:22.802146 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:22.802065    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm_5c965756-5169-45b8-9415-7d226a624cc0/storage-initializer/0.log"
Aug 01 00:34:22.802146 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:22.802102    2546 generic.go:356] "Generic (PLEG): container finished" podID="5c965756-5169-45b8-9415-7d226a624cc0" containerID="54dab054c5f4c24499b71e31a57b8d6677f03616dd5f5e77e7bd540e8900519f" exitCode=1
Aug 01 00:34:22.802591 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:22.802180    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm" event={"ID":"5c965756-5169-45b8-9415-7d226a624cc0","Type":"ContainerDied","Data":"54dab054c5f4c24499b71e31a57b8d6677f03616dd5f5e77e7bd540e8900519f"}
Aug 01 00:34:23.808607 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:23.808579    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm_5c965756-5169-45b8-9415-7d226a624cc0/storage-initializer/0.log"
Aug 01 00:34:23.809055 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:23.808707    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm" event={"ID":"5c965756-5169-45b8-9415-7d226a624cc0","Type":"ContainerStarted","Data":"64b28a2e7f0a1fd2275b7588eeb9e8138602baac96b50c521112c2cecee64014"}
Aug 01 00:34:24.813653 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:24.813625    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm_5c965756-5169-45b8-9415-7d226a624cc0/storage-initializer/1.log"
Aug 01 00:34:24.814021 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:24.814001    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm_5c965756-5169-45b8-9415-7d226a624cc0/storage-initializer/0.log"
Aug 01 00:34:24.814070 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:24.814031    2546 generic.go:356] "Generic (PLEG): container finished" podID="5c965756-5169-45b8-9415-7d226a624cc0" containerID="64b28a2e7f0a1fd2275b7588eeb9e8138602baac96b50c521112c2cecee64014" exitCode=1
Aug 01 00:34:24.814124 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:24.814079    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm" event={"ID":"5c965756-5169-45b8-9415-7d226a624cc0","Type":"ContainerDied","Data":"64b28a2e7f0a1fd2275b7588eeb9e8138602baac96b50c521112c2cecee64014"}
Aug 01 00:34:24.814173 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:24.814143    2546 scope.go:117] "RemoveContainer" containerID="54dab054c5f4c24499b71e31a57b8d6677f03616dd5f5e77e7bd540e8900519f"
Aug 01 00:34:24.814481 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:24.814463    2546 scope.go:117] "RemoveContainer" containerID="54dab054c5f4c24499b71e31a57b8d6677f03616dd5f5e77e7bd540e8900519f"
Aug 01 00:34:24.824383 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:34:24.824363    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm_kserve-ci-e2e-test(5c965756-5169-45b8-9415-7d226a624cc0)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm" podUID="5c965756-5169-45b8-9415-7d226a624cc0"
Aug 01 00:34:24.824543 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:34:24.824521    2546 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm_kserve-ci-e2e-test_5c965756-5169-45b8-9415-7d226a624cc0_0 in pod sandbox 43cfcb885b5337618481aa7dd56b6c078c9653721272b5826c14778b1547d277 from index: no such id: '54dab054c5f4c24499b71e31a57b8d6677f03616dd5f5e77e7bd540e8900519f'" containerID="54dab054c5f4c24499b71e31a57b8d6677f03616dd5f5e77e7bd540e8900519f"
Aug 01 00:34:24.824597 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:24.824549    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54dab054c5f4c24499b71e31a57b8d6677f03616dd5f5e77e7bd540e8900519f"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm_kserve-ci-e2e-test_5c965756-5169-45b8-9415-7d226a624cc0_0 in pod sandbox 43cfcb885b5337618481aa7dd56b6c078c9653721272b5826c14778b1547d277 from index: no such id: '54dab054c5f4c24499b71e31a57b8d6677f03616dd5f5e77e7bd540e8900519f'"
Aug 01 00:34:25.822076 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:25.822044    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm_5c965756-5169-45b8-9415-7d226a624cc0/storage-initializer/1.log"
Aug 01 00:34:28.501632 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.501600    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"]
Aug 01 00:34:28.523719 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.523694    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:28.629676 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.629651    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm_5c965756-5169-45b8-9415-7d226a624cc0/storage-initializer/1.log"
Aug 01 00:34:28.633149 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.633130    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:28.705535 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.705512    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c965756-5169-45b8-9415-7d226a624cc0-kserve-provision-location\") pod \"5c965756-5169-45b8-9415-7d226a624cc0\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") "
Aug 01 00:34:28.705684 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.705545    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkmrw\" (UniqueName: \"kubernetes.io/projected/5c965756-5169-45b8-9415-7d226a624cc0-kube-api-access-pkmrw\") pod \"5c965756-5169-45b8-9415-7d226a624cc0\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") "
Aug 01 00:34:28.705684 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.705570    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c965756-5169-45b8-9415-7d226a624cc0-proxy-tls\") pod \"5c965756-5169-45b8-9415-7d226a624cc0\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") "
Aug 01 00:34:28.705684 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.705626    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c965756-5169-45b8-9415-7d226a624cc0-kube-rbac-proxy-sar-config\") pod \"5c965756-5169-45b8-9415-7d226a624cc0\" (UID: \"5c965756-5169-45b8-9415-7d226a624cc0\") "
Aug 01 00:34:28.705884 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.705770    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c965756-5169-45b8-9415-7d226a624cc0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c965756-5169-45b8-9415-7d226a624cc0" (UID: "5c965756-5169-45b8-9415-7d226a624cc0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:34:28.705990 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.705973    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c965756-5169-45b8-9415-7d226a624cc0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c965756-5169-45b8-9415-7d226a624cc0" (UID: "5c965756-5169-45b8-9415-7d226a624cc0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:34:28.707628 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.707598    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c965756-5169-45b8-9415-7d226a624cc0-kube-api-access-pkmrw" (OuterVolumeSpecName: "kube-api-access-pkmrw") pod "5c965756-5169-45b8-9415-7d226a624cc0" (UID: "5c965756-5169-45b8-9415-7d226a624cc0"). InnerVolumeSpecName "kube-api-access-pkmrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:34:28.707628 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.707614    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c965756-5169-45b8-9415-7d226a624cc0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c965756-5169-45b8-9415-7d226a624cc0" (UID: "5c965756-5169-45b8-9415-7d226a624cc0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:34:28.806526 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.806439    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c965756-5169-45b8-9415-7d226a624cc0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:34:28.806526 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.806471    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c965756-5169-45b8-9415-7d226a624cc0-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:34:28.806526 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.806480    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkmrw\" (UniqueName: \"kubernetes.io/projected/5c965756-5169-45b8-9415-7d226a624cc0-kube-api-access-pkmrw\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:34:28.806526 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.806489    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c965756-5169-45b8-9415-7d226a624cc0-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:34:28.834987 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.834965    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm_5c965756-5169-45b8-9415-7d226a624cc0/storage-initializer/1.log"
Aug 01 00:34:28.835133 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.835040    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm" event={"ID":"5c965756-5169-45b8-9415-7d226a624cc0","Type":"ContainerDied","Data":"43cfcb885b5337618481aa7dd56b6c078c9653721272b5826c14778b1547d277"}
Aug 01 00:34:28.835133 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.835064    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"
Aug 01 00:34:28.835133 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.835109    2546 scope.go:117] "RemoveContainer" containerID="64b28a2e7f0a1fd2275b7588eeb9e8138602baac96b50c521112c2cecee64014"
Aug 01 00:34:28.870718 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.870690    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"]
Aug 01 00:34:28.873975 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:28.873952    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6dc94c8d55-kzhsm"]
Aug 01 00:34:29.613026 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.612997    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:29.631464 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.631437    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"]
Aug 01 00:34:29.631742 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.631731    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kserve-container"
Aug 01 00:34:29.631785 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.631744    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kserve-container"
Aug 01 00:34:29.631785 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.631756    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c965756-5169-45b8-9415-7d226a624cc0" containerName="storage-initializer"
Aug 01 00:34:29.631785 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.631762    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c965756-5169-45b8-9415-7d226a624cc0" containerName="storage-initializer"
Aug 01 00:34:29.631785 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.631785    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="storage-initializer"
Aug 01 00:34:29.631907 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.631791    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="storage-initializer"
Aug 01 00:34:29.631907 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.631798    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c965756-5169-45b8-9415-7d226a624cc0" containerName="storage-initializer"
Aug 01 00:34:29.631907 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.631803    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c965756-5169-45b8-9415-7d226a624cc0" containerName="storage-initializer"
Aug 01 00:34:29.631907 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.631809    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kube-rbac-proxy"
Aug 01 00:34:29.631907 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.631815    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kube-rbac-proxy"
Aug 01 00:34:29.631907 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.631859    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c965756-5169-45b8-9415-7d226a624cc0" containerName="storage-initializer"
Aug 01 00:34:29.631907 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.631867    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c965756-5169-45b8-9415-7d226a624cc0" containerName="storage-initializer"
Aug 01 00:34:29.631907 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.631872    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kserve-container"
Aug 01 00:34:29.631907 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.631879    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8969d00d-dfae-4aae-ab0f-31f4375a6afa" containerName="kube-rbac-proxy"
Aug 01 00:34:29.636316 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.636301    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.639066 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.639043    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Aug 01 00:34:29.639180 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.639073    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 01 00:34:29.639180 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.639101    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 01 00:34:29.639180 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.639129    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Aug 01 00:34:29.639180 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.639176    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 01 00:34:29.640199 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.640182    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-58n7t\""
Aug 01 00:34:29.640336 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.640183    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Aug 01 00:34:29.647772 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.647751    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"]
Aug 01 00:34:29.713875 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.713845    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.714006 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.713883    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stp6t\" (UniqueName: \"kubernetes.io/projected/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kube-api-access-stp6t\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.714006 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.713918    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.714006 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.713974    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.714112 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.714018    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.814522 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.814493    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.814620 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.814550    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.814620 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.814582    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.814620 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.814606    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.814766 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.814629    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stp6t\" (UniqueName: \"kubernetes.io/projected/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kube-api-access-stp6t\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.815014 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.814989    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.815358 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.815337    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.815445 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.815382    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.817024 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.817003    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.823042 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.823023    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stp6t\" (UniqueName: \"kubernetes.io/projected/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kube-api-access-stp6t\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.945817 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.945718    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:29.970695 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:29.970668    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:34:30.078384 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:34:30.076608    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff83942c_bbdb_4a8b_bd0c_2fa94cfb8372.slice/crio-de6fda7ec3369d64c6d65fc8f5436306c02002b4c8894bced4725735a13022c5 WatchSource:0}: Error finding container de6fda7ec3369d64c6d65fc8f5436306c02002b4c8894bced4725735a13022c5: Status 404 returned error can't find the container with id de6fda7ec3369d64c6d65fc8f5436306c02002b4c8894bced4725735a13022c5
Aug 01 00:34:30.081483 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:30.081458    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c965756-5169-45b8-9415-7d226a624cc0" path="/var/lib/kubelet/pods/5c965756-5169-45b8-9415-7d226a624cc0/volumes"
Aug 01 00:34:30.081727 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:30.081714    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"]
Aug 01 00:34:30.845893 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:30.845854    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" event={"ID":"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372","Type":"ContainerStarted","Data":"dfa30588d3d106d9a3ac18dc2595541bfbecd34990d5c953c5aa62bec28ac0c2"}
Aug 01 00:34:30.846312 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:30.845951    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" event={"ID":"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372","Type":"ContainerStarted","Data":"de6fda7ec3369d64c6d65fc8f5436306c02002b4c8894bced4725735a13022c5"}
Aug 01 00:34:31.851928 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:31.851893    2546 generic.go:356] "Generic (PLEG): container finished" podID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerID="dfa30588d3d106d9a3ac18dc2595541bfbecd34990d5c953c5aa62bec28ac0c2" exitCode=0
Aug 01 00:34:31.852308 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:31.851983    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" event={"ID":"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372","Type":"ContainerDied","Data":"dfa30588d3d106d9a3ac18dc2595541bfbecd34990d5c953c5aa62bec28ac0c2"}
Aug 01 00:34:32.858318 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:32.858278    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" event={"ID":"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372","Type":"ContainerStarted","Data":"e6c50c56ce9eb2e5a25aed2b08367537f8139aecd1642062d8fff0a8f876f9e3"}
Aug 01 00:34:32.858787 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:32.858346    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" event={"ID":"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372","Type":"ContainerStarted","Data":"b72a3b9baf4378f4ee09b77efbb88a8b128b16eb1981abb6477a6afae4005c6d"}
Aug 01 00:34:32.858787 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:32.858381    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:32.878415 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:32.878372    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" podStartSLOduration=2.941811266 podStartE2EDuration="3.87835664s" totalImagesPullingTime="0s" totalInitContainerRuntime="936.545374ms" isStatefulPod=false podCreationTimestamp="2026-08-01 00:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:34:32.876737396 +0000 UTC m=+2903.336200271" watchObservedRunningTime="2026-08-01 00:34:32.87835664 +0000 UTC m=+2903.337819453"
Aug 01 00:34:33.864075 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:33.864047    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:33.865117 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:33.865081    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Aug 01 00:34:34.869361 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:34.869314    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Aug 01 00:34:39.874746 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:39.874715    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:34:39.875343 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:39.875316    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Aug 01 00:34:49.875504 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:49.875465    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Aug 01 00:34:59.875372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:34:59.875325    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Aug 01 00:35:09.393386 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:09.393353    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:09.875748 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:09.875709    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Aug 01 00:35:14.543486 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.543454    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:14.550652 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.550626    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"]
Aug 01 00:35:14.555209 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.555193    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.557907 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.557886    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Aug 01 00:35:14.558018 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.557891    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:35:14.558018 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.557997    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Aug 01 00:35:14.567720 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.567694    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"]
Aug 01 00:35:14.595942 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.595913    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73038fa3-74e7-484f-92b2-c4898db1b642-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-dfpn4\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.596113 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.595957    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73038fa3-74e7-484f-92b2-c4898db1b642-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-dfpn4\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.596113 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.596049    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73038fa3-74e7-484f-92b2-c4898db1b642-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-dfpn4\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.596216 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.596130    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzmk2\" (UniqueName: \"kubernetes.io/projected/73038fa3-74e7-484f-92b2-c4898db1b642-kube-api-access-xzmk2\") pod \"isvc-pmml-predictor-8674795f59-dfpn4\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.596216 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.596151    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73038fa3-74e7-484f-92b2-c4898db1b642-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-dfpn4\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.653685 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.653651    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:14.697018 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.696984    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzmk2\" (UniqueName: \"kubernetes.io/projected/73038fa3-74e7-484f-92b2-c4898db1b642-kube-api-access-xzmk2\") pod \"isvc-pmml-predictor-8674795f59-dfpn4\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.697214 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.697026    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73038fa3-74e7-484f-92b2-c4898db1b642-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-dfpn4\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.697214 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.697060    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73038fa3-74e7-484f-92b2-c4898db1b642-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-dfpn4\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.697214 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.697098    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73038fa3-74e7-484f-92b2-c4898db1b642-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-dfpn4\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.697214 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.697146    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73038fa3-74e7-484f-92b2-c4898db1b642-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-dfpn4\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.697504 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.697483    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73038fa3-74e7-484f-92b2-c4898db1b642-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-dfpn4\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.697794 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.697774    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73038fa3-74e7-484f-92b2-c4898db1b642-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-dfpn4\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.697874 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.697804    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73038fa3-74e7-484f-92b2-c4898db1b642-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-dfpn4\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.699578 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.699558    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73038fa3-74e7-484f-92b2-c4898db1b642-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-dfpn4\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.705935 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.705914    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzmk2\" (UniqueName: \"kubernetes.io/projected/73038fa3-74e7-484f-92b2-c4898db1b642-kube-api-access-xzmk2\") pod \"isvc-pmml-predictor-8674795f59-dfpn4\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.864958 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.864926    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:14.997516 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:14.997480    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"]
Aug 01 00:35:15.000773 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:35:15.000746    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73038fa3_74e7_484f_92b2_c4898db1b642.slice/crio-d01db3b9dc1bf59ac92e99674a1433b66373921ad34d15e7045f4fa2d8a57a78 WatchSource:0}: Error finding container d01db3b9dc1bf59ac92e99674a1433b66373921ad34d15e7045f4fa2d8a57a78: Status 404 returned error can't find the container with id d01db3b9dc1bf59ac92e99674a1433b66373921ad34d15e7045f4fa2d8a57a78
Aug 01 00:35:15.046693 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:15.046664    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" event={"ID":"73038fa3-74e7-484f-92b2-c4898db1b642","Type":"ContainerStarted","Data":"d01db3b9dc1bf59ac92e99674a1433b66373921ad34d15e7045f4fa2d8a57a78"}
Aug 01 00:35:16.051890 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:16.051851    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" event={"ID":"73038fa3-74e7-484f-92b2-c4898db1b642","Type":"ContainerStarted","Data":"b84ec605c900033a35e3aeb692e00d1f2a33b56784694037e539a403a74225aa"}
Aug 01 00:35:19.066089 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:19.066003    2546 generic.go:356] "Generic (PLEG): container finished" podID="73038fa3-74e7-484f-92b2-c4898db1b642" containerID="b84ec605c900033a35e3aeb692e00d1f2a33b56784694037e539a403a74225aa" exitCode=0
Aug 01 00:35:19.066089 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:19.066074    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" event={"ID":"73038fa3-74e7-484f-92b2-c4898db1b642","Type":"ContainerDied","Data":"b84ec605c900033a35e3aeb692e00d1f2a33b56784694037e539a403a74225aa"}
Aug 01 00:35:19.876364 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:19.875947    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Aug 01 00:35:26.103268 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:26.103230    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" event={"ID":"73038fa3-74e7-484f-92b2-c4898db1b642","Type":"ContainerStarted","Data":"ea8c263a45f7232346c07edf20fadcc025dc2ecf4f81e4cb21cbeba14c8022a3"}
Aug 01 00:35:27.109507 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:27.109473    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" event={"ID":"73038fa3-74e7-484f-92b2-c4898db1b642","Type":"ContainerStarted","Data":"2475fd7f21f679bbb0a645400acad01bc44e89e2eb134256e3994b1aad35f044"}
Aug 01 00:35:27.109890 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:27.109724    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:27.109890 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:27.109790    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:27.111033 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:27.111009    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Aug 01 00:35:27.129094 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:27.129044    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" podStartSLOduration=2.511155116 podStartE2EDuration="13.129026387s" totalImagesPullingTime="6.94506688s" totalInitContainerRuntime="3.672804391s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:35:27.128582767 +0000 UTC m=+2957.588045580" watchObservedRunningTime="2026-08-01 00:35:27.129026387 +0000 UTC m=+2957.588489201"
Aug 01 00:35:28.113695 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:28.113652    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Aug 01 00:35:29.875330 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:29.875288    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Aug 01 00:35:33.118028 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:33.118000    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:35:33.118503 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:33.118474    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Aug 01 00:35:36.049907 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:36.049859    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:39.875846 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:39.875805    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Aug 01 00:35:43.119362 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:43.119318    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Aug 01 00:35:48.076516 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:48.076487    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:35:49.657742 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:49.657661    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"]
Aug 01 00:35:49.658117 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:49.657938    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kserve-container" containerID="cri-o://b72a3b9baf4378f4ee09b77efbb88a8b128b16eb1981abb6477a6afae4005c6d" gracePeriod=30
Aug 01 00:35:49.658117 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:49.657971    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kube-rbac-proxy" containerID="cri-o://e6c50c56ce9eb2e5a25aed2b08367537f8139aecd1642062d8fff0a8f876f9e3" gracePeriod=30
Aug 01 00:35:49.676646 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:49.676617    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:49.870519 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:49.870473    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Aug 01 00:35:50.202946 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:50.202915    2546 generic.go:356] "Generic (PLEG): container finished" podID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerID="e6c50c56ce9eb2e5a25aed2b08367537f8139aecd1642062d8fff0a8f876f9e3" exitCode=2
Aug 01 00:35:50.203121 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:50.202993    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" event={"ID":"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372","Type":"ContainerDied","Data":"e6c50c56ce9eb2e5a25aed2b08367537f8139aecd1642062d8fff0a8f876f9e3"}
Aug 01 00:35:50.713566 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:50.713524    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:50.964790 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:50.964698    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:35:53.119526 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:53.119484    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Aug 01 00:35:54.403853 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.403825    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:35:54.435382 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.435348    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kube-rbac-proxy-sar-config\") pod \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") "
Aug 01 00:35:54.435540 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.435437    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-proxy-tls\") pod \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") "
Aug 01 00:35:54.435540 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.435489    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stp6t\" (UniqueName: \"kubernetes.io/projected/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kube-api-access-stp6t\") pod \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") "
Aug 01 00:35:54.435540 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.435524    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-cabundle-cert\") pod \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") "
Aug 01 00:35:54.435710 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.435584    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kserve-provision-location\") pod \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\" (UID: \"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372\") "
Aug 01 00:35:54.435853 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.435821    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" (UID: "ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:35:54.435916 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.435855    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" (UID: "ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:35:54.435916 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.435859    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" (UID: "ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:35:54.437692 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.437668    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" (UID: "ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:35:54.437807 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.437706    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kube-api-access-stp6t" (OuterVolumeSpecName: "kube-api-access-stp6t") pod "ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" (UID: "ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372"). InnerVolumeSpecName "kube-api-access-stp6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:35:54.536307 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.536202    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:35:54.536307 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.536245    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:35:54.536307 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.536275    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:35:54.536307 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.536286    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stp6t\" (UniqueName: \"kubernetes.io/projected/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-kube-api-access-stp6t\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:35:54.536307 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:54.536297    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:35:55.224191 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:55.224155    2546 generic.go:356] "Generic (PLEG): container finished" podID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerID="b72a3b9baf4378f4ee09b77efbb88a8b128b16eb1981abb6477a6afae4005c6d" exitCode=0
Aug 01 00:35:55.224378 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:55.224218    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"
Aug 01 00:35:55.224378 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:55.224238    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" event={"ID":"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372","Type":"ContainerDied","Data":"b72a3b9baf4378f4ee09b77efbb88a8b128b16eb1981abb6477a6afae4005c6d"}
Aug 01 00:35:55.224378 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:55.224328    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x" event={"ID":"ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372","Type":"ContainerDied","Data":"de6fda7ec3369d64c6d65fc8f5436306c02002b4c8894bced4725735a13022c5"}
Aug 01 00:35:55.224378 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:55.224365    2546 scope.go:117] "RemoveContainer" containerID="e6c50c56ce9eb2e5a25aed2b08367537f8139aecd1642062d8fff0a8f876f9e3"
Aug 01 00:35:55.233385 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:55.233368    2546 scope.go:117] "RemoveContainer" containerID="b72a3b9baf4378f4ee09b77efbb88a8b128b16eb1981abb6477a6afae4005c6d"
Aug 01 00:35:55.240440 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:55.240423    2546 scope.go:117] "RemoveContainer" containerID="dfa30588d3d106d9a3ac18dc2595541bfbecd34990d5c953c5aa62bec28ac0c2"
Aug 01 00:35:55.245923 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:55.245899    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"]
Aug 01 00:35:55.248196 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:55.248179    2546 scope.go:117] "RemoveContainer" containerID="e6c50c56ce9eb2e5a25aed2b08367537f8139aecd1642062d8fff0a8f876f9e3"
Aug 01 00:35:55.248455 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:35:55.248437    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c50c56ce9eb2e5a25aed2b08367537f8139aecd1642062d8fff0a8f876f9e3\": container with ID starting with e6c50c56ce9eb2e5a25aed2b08367537f8139aecd1642062d8fff0a8f876f9e3 not found: ID does not exist" containerID="e6c50c56ce9eb2e5a25aed2b08367537f8139aecd1642062d8fff0a8f876f9e3"
Aug 01 00:35:55.248507 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:55.248464    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c50c56ce9eb2e5a25aed2b08367537f8139aecd1642062d8fff0a8f876f9e3"} err="failed to get container status \"e6c50c56ce9eb2e5a25aed2b08367537f8139aecd1642062d8fff0a8f876f9e3\": rpc error: code = NotFound desc = could not find container \"e6c50c56ce9eb2e5a25aed2b08367537f8139aecd1642062d8fff0a8f876f9e3\": container with ID starting with e6c50c56ce9eb2e5a25aed2b08367537f8139aecd1642062d8fff0a8f876f9e3 not found: ID does not exist"
Aug 01 00:35:55.248507 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:55.248482    2546 scope.go:117] "RemoveContainer" containerID="b72a3b9baf4378f4ee09b77efbb88a8b128b16eb1981abb6477a6afae4005c6d"
Aug 01 00:35:55.248703 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:35:55.248684    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b72a3b9baf4378f4ee09b77efbb88a8b128b16eb1981abb6477a6afae4005c6d\": container with ID starting with b72a3b9baf4378f4ee09b77efbb88a8b128b16eb1981abb6477a6afae4005c6d not found: ID does not exist" containerID="b72a3b9baf4378f4ee09b77efbb88a8b128b16eb1981abb6477a6afae4005c6d"
Aug 01 00:35:55.248759 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:55.248709    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b72a3b9baf4378f4ee09b77efbb88a8b128b16eb1981abb6477a6afae4005c6d"} err="failed to get container status \"b72a3b9baf4378f4ee09b77efbb88a8b128b16eb1981abb6477a6afae4005c6d\": rpc error: code = NotFound desc = could not find container \"b72a3b9baf4378f4ee09b77efbb88a8b128b16eb1981abb6477a6afae4005c6d\": container with ID starting with b72a3b9baf4378f4ee09b77efbb88a8b128b16eb1981abb6477a6afae4005c6d not found: ID does not exist"
Aug 01 00:35:55.248759 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:55.248726    2546 scope.go:117] "RemoveContainer" containerID="dfa30588d3d106d9a3ac18dc2595541bfbecd34990d5c953c5aa62bec28ac0c2"
Aug 01 00:35:55.249072 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:35:55.249046    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa30588d3d106d9a3ac18dc2595541bfbecd34990d5c953c5aa62bec28ac0c2\": container with ID starting with dfa30588d3d106d9a3ac18dc2595541bfbecd34990d5c953c5aa62bec28ac0c2 not found: ID does not exist" containerID="dfa30588d3d106d9a3ac18dc2595541bfbecd34990d5c953c5aa62bec28ac0c2"
Aug 01 00:35:55.249181 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:55.249094    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa30588d3d106d9a3ac18dc2595541bfbecd34990d5c953c5aa62bec28ac0c2"} err="failed to get container status \"dfa30588d3d106d9a3ac18dc2595541bfbecd34990d5c953c5aa62bec28ac0c2\": rpc error: code = NotFound desc = could not find container \"dfa30588d3d106d9a3ac18dc2595541bfbecd34990d5c953c5aa62bec28ac0c2\": container with ID starting with dfa30588d3d106d9a3ac18dc2595541bfbecd34990d5c953c5aa62bec28ac0c2 not found: ID does not exist"
Aug 01 00:35:55.251216 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:55.251197    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-57d885cfbc-ww64x"]
Aug 01 00:35:56.075827 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:35:56.075794    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" path="/var/lib/kubelet/pods/ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372/volumes"
Aug 01 00:36:00.755503 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:00.755470    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:36:01.813266 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.813230    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:36:01.843389 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.843359    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"]
Aug 01 00:36:01.843769 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.843752    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kube-rbac-proxy"
Aug 01 00:36:01.843852 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.843773    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kube-rbac-proxy"
Aug 01 00:36:01.843852 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.843788    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="storage-initializer"
Aug 01 00:36:01.843852 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.843796    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="storage-initializer"
Aug 01 00:36:01.843852 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.843822    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kserve-container"
Aug 01 00:36:01.843852 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.843830    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kserve-container"
Aug 01 00:36:01.844120 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.843920    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kserve-container"
Aug 01 00:36:01.844120 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.843931    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff83942c-bbdb-4a8b-bd0c-2fa94cfb8372" containerName="kube-rbac-proxy"
Aug 01 00:36:01.871262 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.871225    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"]
Aug 01 00:36:01.871409 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.871400    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:01.873952 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.873925    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Aug 01 00:36:01.874085 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.873958    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Aug 01 00:36:01.874085 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.873981    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Aug 01 00:36:01.887566 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.887547    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bded525e-f324-4f2d-b31b-0aaacbb986de-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:01.887708 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.887582    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bded525e-f324-4f2d-b31b-0aaacbb986de-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:01.887708 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.887605    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bded525e-f324-4f2d-b31b-0aaacbb986de-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:01.887708 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.887630    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bded525e-f324-4f2d-b31b-0aaacbb986de-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:01.887708 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.887665    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lvth\" (UniqueName: \"kubernetes.io/projected/bded525e-f324-4f2d-b31b-0aaacbb986de-kube-api-access-9lvth\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:01.954248 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.954224    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:36:01.988398 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.988376    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bded525e-f324-4f2d-b31b-0aaacbb986de-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:01.988467 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.988406    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bded525e-f324-4f2d-b31b-0aaacbb986de-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:01.988467 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.988424    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvth\" (UniqueName: \"kubernetes.io/projected/bded525e-f324-4f2d-b31b-0aaacbb986de-kube-api-access-9lvth\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:01.988567 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.988484    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bded525e-f324-4f2d-b31b-0aaacbb986de-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:01.988567 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.988509    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bded525e-f324-4f2d-b31b-0aaacbb986de-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:01.988881 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.988863    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bded525e-f324-4f2d-b31b-0aaacbb986de-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:01.989049 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.989031    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bded525e-f324-4f2d-b31b-0aaacbb986de-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:01.989118 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.989097    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bded525e-f324-4f2d-b31b-0aaacbb986de-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:01.990954 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.990938    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bded525e-f324-4f2d-b31b-0aaacbb986de-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:01.996560 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:01.996539    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lvth\" (UniqueName: \"kubernetes.io/projected/bded525e-f324-4f2d-b31b-0aaacbb986de-kube-api-access-9lvth\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:02.181850 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:02.181828    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:02.312134 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:02.312105    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"]
Aug 01 00:36:02.314834 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:36:02.314810    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbded525e_f324_4f2d_b31b_0aaacbb986de.slice/crio-021fbce47c8e38df1d07cac102d630d95b7bdccd8fc3a9dc0af74c448238b586 WatchSource:0}: Error finding container 021fbce47c8e38df1d07cac102d630d95b7bdccd8fc3a9dc0af74c448238b586: Status 404 returned error can't find the container with id 021fbce47c8e38df1d07cac102d630d95b7bdccd8fc3a9dc0af74c448238b586
Aug 01 00:36:03.119072 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:03.119035    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Aug 01 00:36:03.259395 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:03.259360    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" event={"ID":"bded525e-f324-4f2d-b31b-0aaacbb986de","Type":"ContainerStarted","Data":"8630b695a5eb672e7c4e341aa9ef6d4f3cd76f75abc90957829072cf714060dd"}
Aug 01 00:36:03.259563 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:03.259435    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" event={"ID":"bded525e-f324-4f2d-b31b-0aaacbb986de","Type":"ContainerStarted","Data":"021fbce47c8e38df1d07cac102d630d95b7bdccd8fc3a9dc0af74c448238b586"}
Aug 01 00:36:04.265090 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:04.265056    2546 generic.go:356] "Generic (PLEG): container finished" podID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerID="8630b695a5eb672e7c4e341aa9ef6d4f3cd76f75abc90957829072cf714060dd" exitCode=0
Aug 01 00:36:04.265518 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:04.265095    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" event={"ID":"bded525e-f324-4f2d-b31b-0aaacbb986de","Type":"ContainerDied","Data":"8630b695a5eb672e7c4e341aa9ef6d4f3cd76f75abc90957829072cf714060dd"}
Aug 01 00:36:05.271788 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:05.271752    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" event={"ID":"bded525e-f324-4f2d-b31b-0aaacbb986de","Type":"ContainerStarted","Data":"4f2259663c30d9beec087c20b376e6bc51ba4e052e8db0589edf3fda5c17f0eb"}
Aug 01 00:36:05.272191 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:05.271820    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:05.272191 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:05.271847    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" event={"ID":"bded525e-f324-4f2d-b31b-0aaacbb986de","Type":"ContainerStarted","Data":"63f56dcfaa5745338dd3d7525a69c0442ec29dde0d3e3b646a6fced40b3ff05d"}
Aug 01 00:36:05.272191 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:05.271881    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:05.273199 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:05.273172    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Aug 01 00:36:05.290526 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:05.290478    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" podStartSLOduration=3.387013522 podStartE2EDuration="4.290460399s" totalImagesPullingTime="0s" totalInitContainerRuntime="903.446877ms" isStatefulPod=false podCreationTimestamp="2026-08-01 00:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:36:05.289825265 +0000 UTC m=+2995.749288079" watchObservedRunningTime="2026-08-01 00:36:05.290460399 +0000 UTC m=+2995.749923212"
Aug 01 00:36:06.275891 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:06.275851    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Aug 01 00:36:11.280068 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:11.280033    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:36:11.280621 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:11.280595    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Aug 01 00:36:13.118823 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:13.118788    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Aug 01 00:36:21.281558 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:21.281515    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Aug 01 00:36:23.118995 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:23.118958    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Aug 01 00:36:31.281392 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:31.281353    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Aug 01 00:36:33.118824 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:33.118779    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Aug 01 00:36:38.070628 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:38.070585    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Aug 01 00:36:41.280941 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:41.280895    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Aug 01 00:36:48.075372 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:48.075346    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:36:51.280968 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:51.280910    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Aug 01 00:36:55.562390 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:55.562356    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"]
Aug 01 00:36:55.562767 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:55.562628    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kserve-container" containerID="cri-o://ea8c263a45f7232346c07edf20fadcc025dc2ecf4f81e4cb21cbeba14c8022a3" gracePeriod=30
Aug 01 00:36:55.562767 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:55.562634    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kube-rbac-proxy" containerID="cri-o://2475fd7f21f679bbb0a645400acad01bc44e89e2eb134256e3994b1aad35f044" gracePeriod=30
Aug 01 00:36:55.580587 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:55.580561    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:36:56.480953 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:56.480921    2546 generic.go:356] "Generic (PLEG): container finished" podID="73038fa3-74e7-484f-92b2-c4898db1b642" containerID="2475fd7f21f679bbb0a645400acad01bc44e89e2eb134256e3994b1aad35f044" exitCode=2
Aug 01 00:36:56.481128 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:56.481002    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" event={"ID":"73038fa3-74e7-484f-92b2-c4898db1b642","Type":"ContainerDied","Data":"2475fd7f21f679bbb0a645400acad01bc44e89e2eb134256e3994b1aad35f044"}
Aug 01 00:36:58.071520 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.071479    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Aug 01 00:36:58.114084 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.114055    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.66:8643/healthz\": dial tcp 10.133.0.66:8643: connect: connection refused"
Aug 01 00:36:58.491264 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.491223    2546 generic.go:356] "Generic (PLEG): container finished" podID="73038fa3-74e7-484f-92b2-c4898db1b642" containerID="ea8c263a45f7232346c07edf20fadcc025dc2ecf4f81e4cb21cbeba14c8022a3" exitCode=0
Aug 01 00:36:58.491357 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.491288    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" event={"ID":"73038fa3-74e7-484f-92b2-c4898db1b642","Type":"ContainerDied","Data":"ea8c263a45f7232346c07edf20fadcc025dc2ecf4f81e4cb21cbeba14c8022a3"}
Aug 01 00:36:58.602103 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.602081    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:36:58.732745 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.732682    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzmk2\" (UniqueName: \"kubernetes.io/projected/73038fa3-74e7-484f-92b2-c4898db1b642-kube-api-access-xzmk2\") pod \"73038fa3-74e7-484f-92b2-c4898db1b642\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") "
Aug 01 00:36:58.732745 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.732717    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73038fa3-74e7-484f-92b2-c4898db1b642-proxy-tls\") pod \"73038fa3-74e7-484f-92b2-c4898db1b642\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") "
Aug 01 00:36:58.732745 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.732746    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73038fa3-74e7-484f-92b2-c4898db1b642-cabundle-cert\") pod \"73038fa3-74e7-484f-92b2-c4898db1b642\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") "
Aug 01 00:36:58.732958 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.732799    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73038fa3-74e7-484f-92b2-c4898db1b642-kube-rbac-proxy-sar-config\") pod \"73038fa3-74e7-484f-92b2-c4898db1b642\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") "
Aug 01 00:36:58.732958 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.732819    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73038fa3-74e7-484f-92b2-c4898db1b642-kserve-provision-location\") pod \"73038fa3-74e7-484f-92b2-c4898db1b642\" (UID: \"73038fa3-74e7-484f-92b2-c4898db1b642\") "
Aug 01 00:36:58.733201 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.733175    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73038fa3-74e7-484f-92b2-c4898db1b642-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "73038fa3-74e7-484f-92b2-c4898db1b642" (UID: "73038fa3-74e7-484f-92b2-c4898db1b642"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:36:58.733320 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.733179    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73038fa3-74e7-484f-92b2-c4898db1b642-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "73038fa3-74e7-484f-92b2-c4898db1b642" (UID: "73038fa3-74e7-484f-92b2-c4898db1b642"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:36:58.733320 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.733182    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73038fa3-74e7-484f-92b2-c4898db1b642-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "73038fa3-74e7-484f-92b2-c4898db1b642" (UID: "73038fa3-74e7-484f-92b2-c4898db1b642"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:36:58.734824 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.734799    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73038fa3-74e7-484f-92b2-c4898db1b642-kube-api-access-xzmk2" (OuterVolumeSpecName: "kube-api-access-xzmk2") pod "73038fa3-74e7-484f-92b2-c4898db1b642" (UID: "73038fa3-74e7-484f-92b2-c4898db1b642"). InnerVolumeSpecName "kube-api-access-xzmk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:36:58.734916 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.734820    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73038fa3-74e7-484f-92b2-c4898db1b642-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73038fa3-74e7-484f-92b2-c4898db1b642" (UID: "73038fa3-74e7-484f-92b2-c4898db1b642"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:36:58.833452 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.833431    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73038fa3-74e7-484f-92b2-c4898db1b642-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:36:58.833452 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.833451    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73038fa3-74e7-484f-92b2-c4898db1b642-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:36:58.833575 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.833460    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73038fa3-74e7-484f-92b2-c4898db1b642-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:36:58.833575 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.833469    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzmk2\" (UniqueName: \"kubernetes.io/projected/73038fa3-74e7-484f-92b2-c4898db1b642-kube-api-access-xzmk2\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:36:58.833575 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:58.833477    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73038fa3-74e7-484f-92b2-c4898db1b642-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:36:59.496946 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:59.496914    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4" event={"ID":"73038fa3-74e7-484f-92b2-c4898db1b642","Type":"ContainerDied","Data":"d01db3b9dc1bf59ac92e99674a1433b66373921ad34d15e7045f4fa2d8a57a78"}
Aug 01 00:36:59.499283 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:59.496982    2546 scope.go:117] "RemoveContainer" containerID="2475fd7f21f679bbb0a645400acad01bc44e89e2eb134256e3994b1aad35f044"
Aug 01 00:36:59.499283 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:59.496934    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"
Aug 01 00:36:59.506267 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:59.505956    2546 scope.go:117] "RemoveContainer" containerID="ea8c263a45f7232346c07edf20fadcc025dc2ecf4f81e4cb21cbeba14c8022a3"
Aug 01 00:36:59.513223 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:59.513207    2546 scope.go:117] "RemoveContainer" containerID="b84ec605c900033a35e3aeb692e00d1f2a33b56784694037e539a403a74225aa"
Aug 01 00:36:59.521324 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:59.521298    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"]
Aug 01 00:36:59.526198 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:36:59.526177    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-dfpn4"]
Aug 01 00:37:00.075928 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.075898    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" path="/var/lib/kubelet/pods/73038fa3-74e7-484f-92b2-c4898db1b642/volumes"
Aug 01 00:37:00.726756 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.726714    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:37:00.749795 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.749766    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"]
Aug 01 00:37:00.750117 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.750105    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="storage-initializer"
Aug 01 00:37:00.750161 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.750119    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="storage-initializer"
Aug 01 00:37:00.750161 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.750136    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kube-rbac-proxy"
Aug 01 00:37:00.750161 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.750142    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kube-rbac-proxy"
Aug 01 00:37:00.750161 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.750155    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kserve-container"
Aug 01 00:37:00.750161 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.750161    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kserve-container"
Aug 01 00:37:00.750344 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.750205    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kube-rbac-proxy"
Aug 01 00:37:00.750344 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.750215    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="73038fa3-74e7-484f-92b2-c4898db1b642" containerName="kserve-container"
Aug 01 00:37:00.754857 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.754842    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:00.757231 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.757208    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Aug 01 00:37:00.757367 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.757287    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Aug 01 00:37:00.757477 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.757463    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 01 00:37:00.763990 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.763716    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"]
Aug 01 00:37:00.949983 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.949952    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-9gbfz\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:00.950143 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.950005    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-9gbfz\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:00.950143 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.950025    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-9gbfz\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:00.950143 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.950079    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-9gbfz\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:00.950143 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:00.950102    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tttvr\" (UniqueName: \"kubernetes.io/projected/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kube-api-access-tttvr\") pod \"isvc-pmml-runtime-predictor-885fc8777-9gbfz\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:01.037556 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.037485    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:37:01.050409 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.050389    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-9gbfz\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:01.050485 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.050432    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-9gbfz\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:01.050485 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.050450    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-9gbfz\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:01.050554 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.050485    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-9gbfz\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:01.050554 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.050521    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tttvr\" (UniqueName: \"kubernetes.io/projected/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kube-api-access-tttvr\") pod \"isvc-pmml-runtime-predictor-885fc8777-9gbfz\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:01.050972 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.050947    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-9gbfz\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:01.051205 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.051183    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-9gbfz\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:01.051267 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.051183    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-9gbfz\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:01.052995 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.052975    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-9gbfz\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:01.057993 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.057974    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tttvr\" (UniqueName: \"kubernetes.io/projected/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kube-api-access-tttvr\") pod \"isvc-pmml-runtime-predictor-885fc8777-9gbfz\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:01.066845 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.066824    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:01.193755 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.193736    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"]
Aug 01 00:37:01.196022 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:37:01.195994    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b362c5_c017_4f10_9d3d_7f26a38b0c3f.slice/crio-33e35ad8b2f7fba8177afa86da1ae5199bad2928cf92a48be56d6e7f3a4c5be6 WatchSource:0}: Error finding container 33e35ad8b2f7fba8177afa86da1ae5199bad2928cf92a48be56d6e7f3a4c5be6: Status 404 returned error can't find the container with id 33e35ad8b2f7fba8177afa86da1ae5199bad2928cf92a48be56d6e7f3a4c5be6
Aug 01 00:37:01.280904 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.280867    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Aug 01 00:37:01.507431 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.507390    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" event={"ID":"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f","Type":"ContainerStarted","Data":"5a0392501dfbb4861ecf52a0749311ae43b822fcf9267f65aacd1ca276fcaf58"}
Aug 01 00:37:01.507602 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:01.507467    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" event={"ID":"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f","Type":"ContainerStarted","Data":"33e35ad8b2f7fba8177afa86da1ae5199bad2928cf92a48be56d6e7f3a4c5be6"}
Aug 01 00:37:05.526077 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:05.526037    2546 generic.go:356] "Generic (PLEG): container finished" podID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerID="5a0392501dfbb4861ecf52a0749311ae43b822fcf9267f65aacd1ca276fcaf58" exitCode=0
Aug 01 00:37:05.526524 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:05.526108    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" event={"ID":"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f","Type":"ContainerDied","Data":"5a0392501dfbb4861ecf52a0749311ae43b822fcf9267f65aacd1ca276fcaf58"}
Aug 01 00:37:06.532906 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:06.532870    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" event={"ID":"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f","Type":"ContainerStarted","Data":"b2a686ab092cd1d1307b6ddf29f51feb463350204b1e8c4aad28b5dfe453bec3"}
Aug 01 00:37:06.533340 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:06.532935    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" event={"ID":"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f","Type":"ContainerStarted","Data":"f32297ba4517674af13ba001ef48fb6efae50c4a27c37417d7bad820c9dceddb"}
Aug 01 00:37:06.533340 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:06.533131    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:06.551496 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:06.551448    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" podStartSLOduration=3.005659988 podStartE2EDuration="6.551433254s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.545773266s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:37:06.551423637 +0000 UTC m=+3057.010886448" watchObservedRunningTime="2026-08-01 00:37:06.551433254 +0000 UTC m=+3057.010896066"
Aug 01 00:37:07.538521 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:07.538493    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:07.539912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:07.539883    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Aug 01 00:37:08.542759 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:08.542719    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Aug 01 00:37:11.280633 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:11.280594    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Aug 01 00:37:13.548468 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:13.548439    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:37:13.548881 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:13.548855    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Aug 01 00:37:21.281227 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:21.281197    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:37:21.883358 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:21.883319    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"]
Aug 01 00:37:21.883672 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:21.883616    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kserve-container" containerID="cri-o://63f56dcfaa5745338dd3d7525a69c0442ec29dde0d3e3b646a6fced40b3ff05d" gracePeriod=30
Aug 01 00:37:21.883811 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:21.883709    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kube-rbac-proxy" containerID="cri-o://4f2259663c30d9beec087c20b376e6bc51ba4e052e8db0589edf3fda5c17f0eb" gracePeriod=30
Aug 01 00:37:21.897824 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:21.897785    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:37:22.610074 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:22.610030    2546 generic.go:356] "Generic (PLEG): container finished" podID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerID="4f2259663c30d9beec087c20b376e6bc51ba4e052e8db0589edf3fda5c17f0eb" exitCode=2
Aug 01 00:37:22.610460 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:22.610099    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" event={"ID":"bded525e-f324-4f2d-b31b-0aaacbb986de","Type":"ContainerDied","Data":"4f2259663c30d9beec087c20b376e6bc51ba4e052e8db0589edf3fda5c17f0eb"}
Aug 01 00:37:22.984462 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:22.984428    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:37:23.130244 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:23.130210    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:37:23.549325 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:23.549284    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Aug 01 00:37:26.023764 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.023738    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:37:26.036630 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.036611    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lvth\" (UniqueName: \"kubernetes.io/projected/bded525e-f324-4f2d-b31b-0aaacbb986de-kube-api-access-9lvth\") pod \"bded525e-f324-4f2d-b31b-0aaacbb986de\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") "
Aug 01 00:37:26.036760 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.036670    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bded525e-f324-4f2d-b31b-0aaacbb986de-cabundle-cert\") pod \"bded525e-f324-4f2d-b31b-0aaacbb986de\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") "
Aug 01 00:37:26.036760 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.036725    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bded525e-f324-4f2d-b31b-0aaacbb986de-proxy-tls\") pod \"bded525e-f324-4f2d-b31b-0aaacbb986de\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") "
Aug 01 00:37:26.036760 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.036741    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bded525e-f324-4f2d-b31b-0aaacbb986de-kube-rbac-proxy-sar-config\") pod \"bded525e-f324-4f2d-b31b-0aaacbb986de\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") "
Aug 01 00:37:26.036939 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.036778    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bded525e-f324-4f2d-b31b-0aaacbb986de-kserve-provision-location\") pod \"bded525e-f324-4f2d-b31b-0aaacbb986de\" (UID: \"bded525e-f324-4f2d-b31b-0aaacbb986de\") "
Aug 01 00:37:26.037141 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.037121    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bded525e-f324-4f2d-b31b-0aaacbb986de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bded525e-f324-4f2d-b31b-0aaacbb986de" (UID: "bded525e-f324-4f2d-b31b-0aaacbb986de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:37:26.037210 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.037119    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bded525e-f324-4f2d-b31b-0aaacbb986de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bded525e-f324-4f2d-b31b-0aaacbb986de" (UID: "bded525e-f324-4f2d-b31b-0aaacbb986de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:37:26.037210 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.037119    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bded525e-f324-4f2d-b31b-0aaacbb986de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bded525e-f324-4f2d-b31b-0aaacbb986de" (UID: "bded525e-f324-4f2d-b31b-0aaacbb986de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:37:26.038628 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.038604    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bded525e-f324-4f2d-b31b-0aaacbb986de-kube-api-access-9lvth" (OuterVolumeSpecName: "kube-api-access-9lvth") pod "bded525e-f324-4f2d-b31b-0aaacbb986de" (UID: "bded525e-f324-4f2d-b31b-0aaacbb986de"). InnerVolumeSpecName "kube-api-access-9lvth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:37:26.038706 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.038645    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bded525e-f324-4f2d-b31b-0aaacbb986de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bded525e-f324-4f2d-b31b-0aaacbb986de" (UID: "bded525e-f324-4f2d-b31b-0aaacbb986de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:37:26.138505 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.138481    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bded525e-f324-4f2d-b31b-0aaacbb986de-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:37:26.138505 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.138504    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bded525e-f324-4f2d-b31b-0aaacbb986de-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:37:26.138666 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.138512    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bded525e-f324-4f2d-b31b-0aaacbb986de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:37:26.138666 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.138521    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bded525e-f324-4f2d-b31b-0aaacbb986de-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:37:26.138666 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.138530    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lvth\" (UniqueName: \"kubernetes.io/projected/bded525e-f324-4f2d-b31b-0aaacbb986de-kube-api-access-9lvth\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:37:26.630039 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.630000    2546 generic.go:356] "Generic (PLEG): container finished" podID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerID="63f56dcfaa5745338dd3d7525a69c0442ec29dde0d3e3b646a6fced40b3ff05d" exitCode=0
Aug 01 00:37:26.630270 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.630065    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"
Aug 01 00:37:26.630270 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.630077    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" event={"ID":"bded525e-f324-4f2d-b31b-0aaacbb986de","Type":"ContainerDied","Data":"63f56dcfaa5745338dd3d7525a69c0442ec29dde0d3e3b646a6fced40b3ff05d"}
Aug 01 00:37:26.630270 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.630142    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529" event={"ID":"bded525e-f324-4f2d-b31b-0aaacbb986de","Type":"ContainerDied","Data":"021fbce47c8e38df1d07cac102d630d95b7bdccd8fc3a9dc0af74c448238b586"}
Aug 01 00:37:26.630270 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.630174    2546 scope.go:117] "RemoveContainer" containerID="4f2259663c30d9beec087c20b376e6bc51ba4e052e8db0589edf3fda5c17f0eb"
Aug 01 00:37:26.638247 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.638233    2546 scope.go:117] "RemoveContainer" containerID="63f56dcfaa5745338dd3d7525a69c0442ec29dde0d3e3b646a6fced40b3ff05d"
Aug 01 00:37:26.645207 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.645190    2546 scope.go:117] "RemoveContainer" containerID="8630b695a5eb672e7c4e341aa9ef6d4f3cd76f75abc90957829072cf714060dd"
Aug 01 00:37:26.647193 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.647175    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"]
Aug 01 00:37:26.650693 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.650667    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-65fc54497b-kw529"]
Aug 01 00:37:26.652626 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.652612    2546 scope.go:117] "RemoveContainer" containerID="4f2259663c30d9beec087c20b376e6bc51ba4e052e8db0589edf3fda5c17f0eb"
Aug 01 00:37:26.652848 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:37:26.652829    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2259663c30d9beec087c20b376e6bc51ba4e052e8db0589edf3fda5c17f0eb\": container with ID starting with 4f2259663c30d9beec087c20b376e6bc51ba4e052e8db0589edf3fda5c17f0eb not found: ID does not exist" containerID="4f2259663c30d9beec087c20b376e6bc51ba4e052e8db0589edf3fda5c17f0eb"
Aug 01 00:37:26.652908 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.652855    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2259663c30d9beec087c20b376e6bc51ba4e052e8db0589edf3fda5c17f0eb"} err="failed to get container status \"4f2259663c30d9beec087c20b376e6bc51ba4e052e8db0589edf3fda5c17f0eb\": rpc error: code = NotFound desc = could not find container \"4f2259663c30d9beec087c20b376e6bc51ba4e052e8db0589edf3fda5c17f0eb\": container with ID starting with 4f2259663c30d9beec087c20b376e6bc51ba4e052e8db0589edf3fda5c17f0eb not found: ID does not exist"
Aug 01 00:37:26.652908 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.652872    2546 scope.go:117] "RemoveContainer" containerID="63f56dcfaa5745338dd3d7525a69c0442ec29dde0d3e3b646a6fced40b3ff05d"
Aug 01 00:37:26.653091 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:37:26.653073    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f56dcfaa5745338dd3d7525a69c0442ec29dde0d3e3b646a6fced40b3ff05d\": container with ID starting with 63f56dcfaa5745338dd3d7525a69c0442ec29dde0d3e3b646a6fced40b3ff05d not found: ID does not exist" containerID="63f56dcfaa5745338dd3d7525a69c0442ec29dde0d3e3b646a6fced40b3ff05d"
Aug 01 00:37:26.653156 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.653101    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f56dcfaa5745338dd3d7525a69c0442ec29dde0d3e3b646a6fced40b3ff05d"} err="failed to get container status \"63f56dcfaa5745338dd3d7525a69c0442ec29dde0d3e3b646a6fced40b3ff05d\": rpc error: code = NotFound desc = could not find container \"63f56dcfaa5745338dd3d7525a69c0442ec29dde0d3e3b646a6fced40b3ff05d\": container with ID starting with 63f56dcfaa5745338dd3d7525a69c0442ec29dde0d3e3b646a6fced40b3ff05d not found: ID does not exist"
Aug 01 00:37:26.653156 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.653122    2546 scope.go:117] "RemoveContainer" containerID="8630b695a5eb672e7c4e341aa9ef6d4f3cd76f75abc90957829072cf714060dd"
Aug 01 00:37:26.653351 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:37:26.653336    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8630b695a5eb672e7c4e341aa9ef6d4f3cd76f75abc90957829072cf714060dd\": container with ID starting with 8630b695a5eb672e7c4e341aa9ef6d4f3cd76f75abc90957829072cf714060dd not found: ID does not exist" containerID="8630b695a5eb672e7c4e341aa9ef6d4f3cd76f75abc90957829072cf714060dd"
Aug 01 00:37:26.653396 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:26.653355    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8630b695a5eb672e7c4e341aa9ef6d4f3cd76f75abc90957829072cf714060dd"} err="failed to get container status \"8630b695a5eb672e7c4e341aa9ef6d4f3cd76f75abc90957829072cf714060dd\": rpc error: code = NotFound desc = could not find container \"8630b695a5eb672e7c4e341aa9ef6d4f3cd76f75abc90957829072cf714060dd\": container with ID starting with 8630b695a5eb672e7c4e341aa9ef6d4f3cd76f75abc90957829072cf714060dd not found: ID does not exist"
Aug 01 00:37:28.079807 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:28.079778    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" path="/var/lib/kubelet/pods/bded525e-f324-4f2d-b31b-0aaacbb986de/volumes"
Aug 01 00:37:32.985096 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:32.985058    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:37:33.549369 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:33.549333    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Aug 01 00:37:43.549535 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:43.549494    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Aug 01 00:37:53.549451 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:37:53.549410    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Aug 01 00:38:03.549459 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:03.549419    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Aug 01 00:38:13.549854 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:13.549808    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Aug 01 00:38:23.550053 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:23.550023    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:38:31.859298 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:31.859248    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"]
Aug 01 00:38:31.859763 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:31.859511    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kserve-container" containerID="cri-o://f32297ba4517674af13ba001ef48fb6efae50c4a27c37417d7bad820c9dceddb" gracePeriod=30
Aug 01 00:38:31.859763 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:31.859542    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kube-rbac-proxy" containerID="cri-o://b2a686ab092cd1d1307b6ddf29f51feb463350204b1e8c4aad28b5dfe453bec3" gracePeriod=30
Aug 01 00:38:31.879279 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:31.879232    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:32.901429 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:32.901398    2546 generic.go:356] "Generic (PLEG): container finished" podID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerID="b2a686ab092cd1d1307b6ddf29f51feb463350204b1e8c4aad28b5dfe453bec3" exitCode=2
Aug 01 00:38:32.901773 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:32.901436    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" event={"ID":"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f","Type":"ContainerDied","Data":"b2a686ab092cd1d1307b6ddf29f51feb463350204b1e8c4aad28b5dfe453bec3"}
Aug 01 00:38:33.543590 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:33.543546    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.68:8643/healthz\": dial tcp 10.133.0.68:8643: connect: connection refused"
Aug 01 00:38:33.548916 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:33.548891    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Aug 01 00:38:34.896463 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.896436    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:38:34.910237 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.910213    2546 generic.go:356] "Generic (PLEG): container finished" podID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerID="f32297ba4517674af13ba001ef48fb6efae50c4a27c37417d7bad820c9dceddb" exitCode=0
Aug 01 00:38:34.910347 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.910248    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" event={"ID":"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f","Type":"ContainerDied","Data":"f32297ba4517674af13ba001ef48fb6efae50c4a27c37417d7bad820c9dceddb"}
Aug 01 00:38:34.910347 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.910268    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"
Aug 01 00:38:34.910347 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.910335    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz" event={"ID":"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f","Type":"ContainerDied","Data":"33e35ad8b2f7fba8177afa86da1ae5199bad2928cf92a48be56d6e7f3a4c5be6"}
Aug 01 00:38:34.910449 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.910361    2546 scope.go:117] "RemoveContainer" containerID="b2a686ab092cd1d1307b6ddf29f51feb463350204b1e8c4aad28b5dfe453bec3"
Aug 01 00:38:34.919790 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.919768    2546 scope.go:117] "RemoveContainer" containerID="f32297ba4517674af13ba001ef48fb6efae50c4a27c37417d7bad820c9dceddb"
Aug 01 00:38:34.927072 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.927050    2546 scope.go:117] "RemoveContainer" containerID="5a0392501dfbb4861ecf52a0749311ae43b822fcf9267f65aacd1ca276fcaf58"
Aug 01 00:38:34.934347 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.934323    2546 scope.go:117] "RemoveContainer" containerID="b2a686ab092cd1d1307b6ddf29f51feb463350204b1e8c4aad28b5dfe453bec3"
Aug 01 00:38:34.934616 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:38:34.934595    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a686ab092cd1d1307b6ddf29f51feb463350204b1e8c4aad28b5dfe453bec3\": container with ID starting with b2a686ab092cd1d1307b6ddf29f51feb463350204b1e8c4aad28b5dfe453bec3 not found: ID does not exist" containerID="b2a686ab092cd1d1307b6ddf29f51feb463350204b1e8c4aad28b5dfe453bec3"
Aug 01 00:38:34.934707 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.934623    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a686ab092cd1d1307b6ddf29f51feb463350204b1e8c4aad28b5dfe453bec3"} err="failed to get container status \"b2a686ab092cd1d1307b6ddf29f51feb463350204b1e8c4aad28b5dfe453bec3\": rpc error: code = NotFound desc = could not find container \"b2a686ab092cd1d1307b6ddf29f51feb463350204b1e8c4aad28b5dfe453bec3\": container with ID starting with b2a686ab092cd1d1307b6ddf29f51feb463350204b1e8c4aad28b5dfe453bec3 not found: ID does not exist"
Aug 01 00:38:34.934707 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.934641    2546 scope.go:117] "RemoveContainer" containerID="f32297ba4517674af13ba001ef48fb6efae50c4a27c37417d7bad820c9dceddb"
Aug 01 00:38:34.934873 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:38:34.934855    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f32297ba4517674af13ba001ef48fb6efae50c4a27c37417d7bad820c9dceddb\": container with ID starting with f32297ba4517674af13ba001ef48fb6efae50c4a27c37417d7bad820c9dceddb not found: ID does not exist" containerID="f32297ba4517674af13ba001ef48fb6efae50c4a27c37417d7bad820c9dceddb"
Aug 01 00:38:34.934940 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.934879    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32297ba4517674af13ba001ef48fb6efae50c4a27c37417d7bad820c9dceddb"} err="failed to get container status \"f32297ba4517674af13ba001ef48fb6efae50c4a27c37417d7bad820c9dceddb\": rpc error: code = NotFound desc = could not find container \"f32297ba4517674af13ba001ef48fb6efae50c4a27c37417d7bad820c9dceddb\": container with ID starting with f32297ba4517674af13ba001ef48fb6efae50c4a27c37417d7bad820c9dceddb not found: ID does not exist"
Aug 01 00:38:34.934940 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.934896    2546 scope.go:117] "RemoveContainer" containerID="5a0392501dfbb4861ecf52a0749311ae43b822fcf9267f65aacd1ca276fcaf58"
Aug 01 00:38:34.935121 ip-10-0-140-138 kubenswrapper[2546]: E0801 00:38:34.935101    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0392501dfbb4861ecf52a0749311ae43b822fcf9267f65aacd1ca276fcaf58\": container with ID starting with 5a0392501dfbb4861ecf52a0749311ae43b822fcf9267f65aacd1ca276fcaf58 not found: ID does not exist" containerID="5a0392501dfbb4861ecf52a0749311ae43b822fcf9267f65aacd1ca276fcaf58"
Aug 01 00:38:34.935189 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.935123    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0392501dfbb4861ecf52a0749311ae43b822fcf9267f65aacd1ca276fcaf58"} err="failed to get container status \"5a0392501dfbb4861ecf52a0749311ae43b822fcf9267f65aacd1ca276fcaf58\": rpc error: code = NotFound desc = could not find container \"5a0392501dfbb4861ecf52a0749311ae43b822fcf9267f65aacd1ca276fcaf58\": container with ID starting with 5a0392501dfbb4861ecf52a0749311ae43b822fcf9267f65aacd1ca276fcaf58 not found: ID does not exist"
Aug 01 00:38:34.986022 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.985972    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kube-rbac-proxy-sar-config\") pod \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") "
Aug 01 00:38:34.986022 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.986020    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kserve-provision-location\") pod \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") "
Aug 01 00:38:34.986138 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.986042    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-proxy-tls\") pod \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") "
Aug 01 00:38:34.986175 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.986161    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-cabundle-cert\") pod \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") "
Aug 01 00:38:34.986219 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.986203    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tttvr\" (UniqueName: \"kubernetes.io/projected/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kube-api-access-tttvr\") pod \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\" (UID: \"a4b362c5-c017-4f10-9d3d-7f26a38b0c3f\") "
Aug 01 00:38:34.986348 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.986322    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" (UID: "a4b362c5-c017-4f10-9d3d-7f26a38b0c3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 01 00:38:34.986401 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.986354    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" (UID: "a4b362c5-c017-4f10-9d3d-7f26a38b0c3f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:38:34.986524 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.986506    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" (UID: "a4b362c5-c017-4f10-9d3d-7f26a38b0c3f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 01 00:38:34.986524 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.986513    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:38:34.986648 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.986536    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kserve-provision-location\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:38:34.987965 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.987948    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" (UID: "a4b362c5-c017-4f10-9d3d-7f26a38b0c3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 01 00:38:34.988163 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:34.988148    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kube-api-access-tttvr" (OuterVolumeSpecName: "kube-api-access-tttvr") pod "a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" (UID: "a4b362c5-c017-4f10-9d3d-7f26a38b0c3f"). InnerVolumeSpecName "kube-api-access-tttvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 01 00:38:35.087677 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:35.087647    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-proxy-tls\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:38:35.087677 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:35.087677    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-cabundle-cert\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:38:35.087835 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:35.087690    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tttvr\" (UniqueName: \"kubernetes.io/projected/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f-kube-api-access-tttvr\") on node \"ip-10-0-140-138.ec2.internal\" DevicePath \"\""
Aug 01 00:38:35.231814 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:35.231783    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"]
Aug 01 00:38:35.236944 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:35.236891    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-9gbfz"]
Aug 01 00:38:36.074856 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:36.074822    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" path="/var/lib/kubelet/pods/a4b362c5-c017-4f10-9d3d-7f26a38b0c3f/volumes"
Aug 01 00:38:38.431070 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:38.431039    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:38.492318 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:38.492292    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:38:53.797079 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:38:53.797045    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:39:12.379494 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.379459    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:39:12.535807 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.535779    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7jcv6/must-gather-b45zp"]
Aug 01 00:39:12.536090 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536079    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kube-rbac-proxy"
Aug 01 00:39:12.536132 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536092    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kube-rbac-proxy"
Aug 01 00:39:12.536132 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536101    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kserve-container"
Aug 01 00:39:12.536132 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536107    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kserve-container"
Aug 01 00:39:12.536132 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536122    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="storage-initializer"
Aug 01 00:39:12.536132 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536128    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="storage-initializer"
Aug 01 00:39:12.536306 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536135    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="storage-initializer"
Aug 01 00:39:12.536306 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536140    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="storage-initializer"
Aug 01 00:39:12.536306 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536148    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kserve-container"
Aug 01 00:39:12.536306 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536153    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kserve-container"
Aug 01 00:39:12.536306 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536163    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kube-rbac-proxy"
Aug 01 00:39:12.536306 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536168    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kube-rbac-proxy"
Aug 01 00:39:12.536306 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536215    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kube-rbac-proxy"
Aug 01 00:39:12.536306 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536222    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4b362c5-c017-4f10-9d3d-7f26a38b0c3f" containerName="kserve-container"
Aug 01 00:39:12.536306 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536247    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kserve-container"
Aug 01 00:39:12.536306 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.536266    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bded525e-f324-4f2d-b31b-0aaacbb986de" containerName="kube-rbac-proxy"
Aug 01 00:39:12.537728 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.537705    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:39:12.540373 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.540359    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7jcv6/must-gather-b45zp"
Aug 01 00:39:12.543046 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.543024    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7jcv6\"/\"openshift-service-ca.crt\""
Aug 01 00:39:12.543163 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.543057    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7jcv6\"/\"kube-root-ca.crt\""
Aug 01 00:39:12.543984 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.543971    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7jcv6\"/\"default-dockercfg-mwjjj\""
Aug 01 00:39:12.546664 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.546644    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7jcv6/must-gather-b45zp"]
Aug 01 00:39:12.596982 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.596946    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:39:12.686915 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.686860    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57d94\" (UniqueName: \"kubernetes.io/projected/e6858d1f-b7be-4598-a767-d5fb24dbbbf4-kube-api-access-57d94\") pod \"must-gather-b45zp\" (UID: \"e6858d1f-b7be-4598-a767-d5fb24dbbbf4\") " pod="openshift-must-gather-7jcv6/must-gather-b45zp"
Aug 01 00:39:12.686915 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.686905    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e6858d1f-b7be-4598-a767-d5fb24dbbbf4-must-gather-output\") pod \"must-gather-b45zp\" (UID: \"e6858d1f-b7be-4598-a767-d5fb24dbbbf4\") " pod="openshift-must-gather-7jcv6/must-gather-b45zp"
Aug 01 00:39:12.787571 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.787549    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57d94\" (UniqueName: \"kubernetes.io/projected/e6858d1f-b7be-4598-a767-d5fb24dbbbf4-kube-api-access-57d94\") pod \"must-gather-b45zp\" (UID: \"e6858d1f-b7be-4598-a767-d5fb24dbbbf4\") " pod="openshift-must-gather-7jcv6/must-gather-b45zp"
Aug 01 00:39:12.787653 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.787585    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e6858d1f-b7be-4598-a767-d5fb24dbbbf4-must-gather-output\") pod \"must-gather-b45zp\" (UID: \"e6858d1f-b7be-4598-a767-d5fb24dbbbf4\") " pod="openshift-must-gather-7jcv6/must-gather-b45zp"
Aug 01 00:39:12.787851 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.787837    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e6858d1f-b7be-4598-a767-d5fb24dbbbf4-must-gather-output\") pod \"must-gather-b45zp\" (UID: \"e6858d1f-b7be-4598-a767-d5fb24dbbbf4\") " pod="openshift-must-gather-7jcv6/must-gather-b45zp"
Aug 01 00:39:12.795186 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.795164    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57d94\" (UniqueName: \"kubernetes.io/projected/e6858d1f-b7be-4598-a767-d5fb24dbbbf4-kube-api-access-57d94\") pod \"must-gather-b45zp\" (UID: \"e6858d1f-b7be-4598-a767-d5fb24dbbbf4\") " pod="openshift-must-gather-7jcv6/must-gather-b45zp"
Aug 01 00:39:12.857335 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.857316    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7jcv6/must-gather-b45zp"
Aug 01 00:39:12.985154 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.985129    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7jcv6/must-gather-b45zp"]
Aug 01 00:39:12.987400 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:39:12.987362    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6858d1f_b7be_4598_a767_d5fb24dbbbf4.slice/crio-2d28a1476d8c5040347a0f96679f9c8d7617fc13581910cd2dda395d1b7dfe41 WatchSource:0}: Error finding container 2d28a1476d8c5040347a0f96679f9c8d7617fc13581910cd2dda395d1b7dfe41: Status 404 returned error can't find the container with id 2d28a1476d8c5040347a0f96679f9c8d7617fc13581910cd2dda395d1b7dfe41
Aug 01 00:39:12.989162 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:12.989147    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 01 00:39:13.060616 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:13.060585    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7jcv6/must-gather-b45zp" event={"ID":"e6858d1f-b7be-4598-a767-d5fb24dbbbf4","Type":"ContainerStarted","Data":"2d28a1476d8c5040347a0f96679f9c8d7617fc13581910cd2dda395d1b7dfe41"}
Aug 01 00:39:14.066912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:14.066847    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7jcv6/must-gather-b45zp" event={"ID":"e6858d1f-b7be-4598-a767-d5fb24dbbbf4","Type":"ContainerStarted","Data":"4fc16803541ffe2051145ab1a8f6199fbe04e48c4b96cd67273f8bacb7795af8"}
Aug 01 00:39:14.066912 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:14.066892    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7jcv6/must-gather-b45zp" event={"ID":"e6858d1f-b7be-4598-a767-d5fb24dbbbf4","Type":"ContainerStarted","Data":"10cfb17ba3b3d9626af05111ad0ea78f2bf5c552b413da3bd20e4bea27bc3f17"}
Aug 01 00:39:14.083726 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:14.083685    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7jcv6/must-gather-b45zp" podStartSLOduration=1.283341695 podStartE2EDuration="2.083671428s" totalImagesPullingTime="800.329733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-01 00:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:39:14.081986022 +0000 UTC m=+3184.541448838" watchObservedRunningTime="2026-08-01 00:39:14.083671428 +0000 UTC m=+3184.543134241"
Aug 01 00:39:15.344463 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:15.344430    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zmcw2_4eb3a0b2-c573-43c7-a8dd-a01c3545d78b/global-pull-secret-syncer/0.log"
Aug 01 00:39:15.395527 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:15.395502    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log"
Aug 01 00:39:15.517001 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:15.516974    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-138.ec2.internal_d1004082fdc42266a4ca7100481dac3f/haproxy/0.log"
Aug 01 00:39:18.008945 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:18.008885    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:39:18.105816 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:18.105764    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdpjh_6385e683-4bac-448e-b72d-240f54800c50/konnectivity-agent/0.log" limit=1024
Aug 01 00:39:19.379974 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:19.379940    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_425aeefe-ddeb-46e9-809d-5563e37b13f0/alertmanager/0.log"
Aug 01 00:39:19.411344 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:19.411121    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_425aeefe-ddeb-46e9-809d-5563e37b13f0/config-reloader/0.log"
Aug 01 00:39:19.444417 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:19.444365    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_425aeefe-ddeb-46e9-809d-5563e37b13f0/kube-rbac-proxy-web/0.log"
Aug 01 00:39:19.475527 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:19.475501    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_425aeefe-ddeb-46e9-809d-5563e37b13f0/kube-rbac-proxy/0.log"
Aug 01 00:39:19.511418 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:19.511394    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_425aeefe-ddeb-46e9-809d-5563e37b13f0/kube-rbac-proxy-metric/0.log"
Aug 01 00:39:19.535091 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:19.535066    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_425aeefe-ddeb-46e9-809d-5563e37b13f0/prom-label-proxy/0.log"
Aug 01 00:39:19.559826 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:19.559766    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_425aeefe-ddeb-46e9-809d-5563e37b13f0/init-config-reloader/0.log"
Aug 01 00:39:19.606857 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:19.606823    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-c8dsw_81b34b10-cd40-4782-b31e-086b1aa66312/cluster-monitoring-operator/0.log"
Aug 01 00:39:19.728964 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:19.728891    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6dc885d577-728pn_d286d217-c582-4af1-814c-efe3f56cf7f5/metrics-server/0.log"
Aug 01 00:39:19.884284 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:19.884246    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bv2b6_35b7e250-b205-4ec0-aa2a-32b283a15e8b/node-exporter/0.log"
Aug 01 00:39:19.945896 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:19.945860    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bv2b6_35b7e250-b205-4ec0-aa2a-32b283a15e8b/kube-rbac-proxy/0.log"
Aug 01 00:39:19.971943 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:19.971905    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bv2b6_35b7e250-b205-4ec0-aa2a-32b283a15e8b/init-textfile/0.log"
Aug 01 00:39:20.470200 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:20.470166    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86669fc994-n7lns_81065d77-f670-4ddc-a06a-04a88c98fc8e/thanos-query/0.log"
Aug 01 00:39:20.492440 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:20.492408    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86669fc994-n7lns_81065d77-f670-4ddc-a06a-04a88c98fc8e/kube-rbac-proxy-web/0.log"
Aug 01 00:39:20.511507 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:20.511479    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86669fc994-n7lns_81065d77-f670-4ddc-a06a-04a88c98fc8e/kube-rbac-proxy/0.log"
Aug 01 00:39:20.529450 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:20.529421    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86669fc994-n7lns_81065d77-f670-4ddc-a06a-04a88c98fc8e/prom-label-proxy/0.log"
Aug 01 00:39:20.546457 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:20.546434    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86669fc994-n7lns_81065d77-f670-4ddc-a06a-04a88c98fc8e/kube-rbac-proxy-rules/0.log"
Aug 01 00:39:20.569178 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:20.569141    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86669fc994-n7lns_81065d77-f670-4ddc-a06a-04a88c98fc8e/kube-rbac-proxy-metrics/0.log"
Aug 01 00:39:22.785843 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:22.785817    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-776868b596-65p6t_fce27230-d438-4930-afb1-f0bc442dbbff/console/0.log"
Aug 01 00:39:22.817113 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:22.817080    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-wl8v7_7ed00006-4005-410f-b5f0-cc72951ee8b4/download-server/0.log"
Aug 01 00:39:23.705398 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.705365    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"]
Aug 01 00:39:23.711068 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.711040    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:23.717847 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.717824    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"]
Aug 01 00:39:23.886566 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.886529    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/48878d7d-ef44-4656-8880-18a8f2ce512a-podres\") pod \"perf-node-gather-daemonset-hcxgm\" (UID: \"48878d7d-ef44-4656-8880-18a8f2ce512a\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:23.887066 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.886586    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwcg5\" (UniqueName: \"kubernetes.io/projected/48878d7d-ef44-4656-8880-18a8f2ce512a-kube-api-access-mwcg5\") pod \"perf-node-gather-daemonset-hcxgm\" (UID: \"48878d7d-ef44-4656-8880-18a8f2ce512a\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:23.887066 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.886626    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48878d7d-ef44-4656-8880-18a8f2ce512a-sys\") pod \"perf-node-gather-daemonset-hcxgm\" (UID: \"48878d7d-ef44-4656-8880-18a8f2ce512a\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:23.887066 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.886658    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/48878d7d-ef44-4656-8880-18a8f2ce512a-proc\") pod \"perf-node-gather-daemonset-hcxgm\" (UID: \"48878d7d-ef44-4656-8880-18a8f2ce512a\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:23.887066 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.886694    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48878d7d-ef44-4656-8880-18a8f2ce512a-lib-modules\") pod \"perf-node-gather-daemonset-hcxgm\" (UID: \"48878d7d-ef44-4656-8880-18a8f2ce512a\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:23.987393 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.987314    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/48878d7d-ef44-4656-8880-18a8f2ce512a-proc\") pod \"perf-node-gather-daemonset-hcxgm\" (UID: \"48878d7d-ef44-4656-8880-18a8f2ce512a\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:23.987534 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.987393    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48878d7d-ef44-4656-8880-18a8f2ce512a-lib-modules\") pod \"perf-node-gather-daemonset-hcxgm\" (UID: \"48878d7d-ef44-4656-8880-18a8f2ce512a\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:23.987534 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.987432    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/48878d7d-ef44-4656-8880-18a8f2ce512a-proc\") pod \"perf-node-gather-daemonset-hcxgm\" (UID: \"48878d7d-ef44-4656-8880-18a8f2ce512a\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:23.987534 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.987445    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/48878d7d-ef44-4656-8880-18a8f2ce512a-podres\") pod \"perf-node-gather-daemonset-hcxgm\" (UID: \"48878d7d-ef44-4656-8880-18a8f2ce512a\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:23.987534 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.987511    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwcg5\" (UniqueName: \"kubernetes.io/projected/48878d7d-ef44-4656-8880-18a8f2ce512a-kube-api-access-mwcg5\") pod \"perf-node-gather-daemonset-hcxgm\" (UID: \"48878d7d-ef44-4656-8880-18a8f2ce512a\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:23.987691 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.987549    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48878d7d-ef44-4656-8880-18a8f2ce512a-sys\") pod \"perf-node-gather-daemonset-hcxgm\" (UID: \"48878d7d-ef44-4656-8880-18a8f2ce512a\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:23.987691 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.987558    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48878d7d-ef44-4656-8880-18a8f2ce512a-lib-modules\") pod \"perf-node-gather-daemonset-hcxgm\" (UID: \"48878d7d-ef44-4656-8880-18a8f2ce512a\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:23.987691 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.987563    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/48878d7d-ef44-4656-8880-18a8f2ce512a-podres\") pod \"perf-node-gather-daemonset-hcxgm\" (UID: \"48878d7d-ef44-4656-8880-18a8f2ce512a\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:23.987691 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.987681    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48878d7d-ef44-4656-8880-18a8f2ce512a-sys\") pod \"perf-node-gather-daemonset-hcxgm\" (UID: \"48878d7d-ef44-4656-8880-18a8f2ce512a\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:23.996706 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:23.996677    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwcg5\" (UniqueName: \"kubernetes.io/projected/48878d7d-ef44-4656-8880-18a8f2ce512a-kube-api-access-mwcg5\") pod \"perf-node-gather-daemonset-hcxgm\" (UID: \"48878d7d-ef44-4656-8880-18a8f2ce512a\") " pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:24.025422 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:24.025379    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:24.120983 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:24.120941    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jw8jw_fec5df48-6c8a-49ff-bb01-43d1d87ca8fd/dns/0.log"
Aug 01 00:39:24.139199 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:24.139177    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jw8jw_fec5df48-6c8a-49ff-bb01-43d1d87ca8fd/kube-rbac-proxy/0.log"
Aug 01 00:39:24.177527 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:24.177498    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"]
Aug 01 00:39:24.181795 ip-10-0-140-138 kubenswrapper[2546]: W0801 00:39:24.181747    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod48878d7d_ef44_4656_8880_18a8f2ce512a.slice/crio-427fac7b9e2681f683413916d3332966d48ae6d5e5f105cb712617024f36d006 WatchSource:0}: Error finding container 427fac7b9e2681f683413916d3332966d48ae6d5e5f105cb712617024f36d006: Status 404 returned error can't find the container with id 427fac7b9e2681f683413916d3332966d48ae6d5e5f105cb712617024f36d006
Aug 01 00:39:24.271766 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:24.271687    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v58mz_655fd51c-e55e-4fa9-8a11-8de5f622b670/dns-node-resolver/0.log"
Aug 01 00:39:24.719104 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:24.719069    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65d74d959b-7g6nj_fbab1df5-84fb-437a-8b11-6f04726b5b2e/registry/0.log"
Aug 01 00:39:24.736686 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:24.736655    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hp4zj_5278364c-d649-48ba-8388-d277b82f21a7/node-ca/0.log"
Aug 01 00:39:25.133511 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:25.133466    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm" event={"ID":"48878d7d-ef44-4656-8880-18a8f2ce512a","Type":"ContainerStarted","Data":"39b8502f3af7ca7b666a01a5593c2c1dd75c8ea33fa26adf38fa866ba2a12c4c"}
Aug 01 00:39:25.133981 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:25.133592    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm" event={"ID":"48878d7d-ef44-4656-8880-18a8f2ce512a","Type":"ContainerStarted","Data":"427fac7b9e2681f683413916d3332966d48ae6d5e5f105cb712617024f36d006"}
Aug 01 00:39:25.133981 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:25.133765    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:25.149200 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:25.149149    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm" podStartSLOduration=2.149119983 podStartE2EDuration="2.149119983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-01 00:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-01 00:39:25.147301055 +0000 UTC m=+3195.606763885" watchObservedRunningTime="2026-08-01 00:39:25.149119983 +0000 UTC m=+3195.608582795"
Aug 01 00:39:25.480084 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:25.480011    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f44c6f5f8-5hkk9_e005be7e-fed7-468e-9a7d-601db46f362f/router/0.log"
Aug 01 00:39:25.771903 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:25.771831    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7m7vf_447bf0cf-4705-4db9-a4ba-cc6d257d3170/serve-healthcheck-canary/0.log"
Aug 01 00:39:26.192424 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:26.192393    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jkssq_eefe175d-a2df-4722-99af-40719128fc4d/kube-rbac-proxy/0.log"
Aug 01 00:39:26.207805 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:26.207780    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jkssq_eefe175d-a2df-4722-99af-40719128fc4d/exporter/0.log"
Aug 01 00:39:26.224060 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:26.224043    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jkssq_eefe175d-a2df-4722-99af-40719128fc4d/extractor/0.log"
Aug 01 00:39:28.427383 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:28.427357    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-j7bld_f369f278-97a5-4457-b5f1-db1f8aca9940/manager/0.log"
Aug 01 00:39:28.442377 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:28.442355    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-g74kd_ed88cc15-1b2d-4554-93e2-5b8b01298e6b/server/0.log"
Aug 01 00:39:31.149735 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:31.149707    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7jcv6/perf-node-gather-daemonset-hcxgm"
Aug 01 00:39:33.662264 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:33.662222    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4q4d7_a66649bd-a671-4bee-af65-7930bfc49e55/kube-multus-additional-cni-plugins/0.log"
Aug 01 00:39:33.691097 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:33.691072    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4q4d7_a66649bd-a671-4bee-af65-7930bfc49e55/egress-router-binary-copy/0.log"
Aug 01 00:39:33.707303 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:33.707283    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4q4d7_a66649bd-a671-4bee-af65-7930bfc49e55/cni-plugins/0.log"
Aug 01 00:39:33.787598 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:33.787576    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4q4d7_a66649bd-a671-4bee-af65-7930bfc49e55/bond-cni-plugin/0.log"
Aug 01 00:39:33.801119 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:33.801099    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4q4d7_a66649bd-a671-4bee-af65-7930bfc49e55/routeoverride-cni/0.log"
Aug 01 00:39:33.817066 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:33.817047    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4q4d7_a66649bd-a671-4bee-af65-7930bfc49e55/whereabouts-cni-bincopy/0.log"
Aug 01 00:39:33.832537 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:33.832519    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4q4d7_a66649bd-a671-4bee-af65-7930bfc49e55/whereabouts-cni/0.log"
Aug 01 00:39:34.131943 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:34.131894    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjsqg_45223f8f-c4dd-4b50-9c18-b182fc6e3fb7/kube-multus/0.log"
Aug 01 00:39:34.181302 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:34.181274    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5qcqv_1fcbea3e-f254-440f-9281-bb84b6c5afaf/network-metrics-daemon/0.log"
Aug 01 00:39:34.194665 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:34.194632    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5qcqv_1fcbea3e-f254-440f-9281-bb84b6c5afaf/kube-rbac-proxy/0.log"
Aug 01 00:39:35.033975 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:35.033944    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qmq9_f883b5d2-c71f-40f7-bdbd-131e956f4cef/ovn-controller/0.log"
Aug 01 00:39:35.049737 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:35.049712    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qmq9_f883b5d2-c71f-40f7-bdbd-131e956f4cef/ovn-acl-logging/0.log"
Aug 01 00:39:35.070701 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:35.070684    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qmq9_f883b5d2-c71f-40f7-bdbd-131e956f4cef/kube-rbac-proxy-node/0.log"
Aug 01 00:39:35.088641 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:35.088622    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qmq9_f883b5d2-c71f-40f7-bdbd-131e956f4cef/kube-rbac-proxy-ovn-metrics/0.log"
Aug 01 00:39:35.104485 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:35.104460    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qmq9_f883b5d2-c71f-40f7-bdbd-131e956f4cef/northd/0.log"
Aug 01 00:39:35.122633 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:35.122615    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qmq9_f883b5d2-c71f-40f7-bdbd-131e956f4cef/nbdb/0.log"
Aug 01 00:39:35.138904 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:35.138888    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qmq9_f883b5d2-c71f-40f7-bdbd-131e956f4cef/sbdb/0.log"
Aug 01 00:39:35.179819 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:35.179794    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2qmq9_f883b5d2-c71f-40f7-bdbd-131e956f4cef/ovnkube-controller/0.log"
Aug 01 00:39:36.588834 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:36.588805    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-65bdbbff45-ktsf2_8c85d11a-d782-4c52-9126-d9b563ed453f/check-endpoints/0.log"
Aug 01 00:39:36.649094 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:36.649068    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-wr59z_4c5d6736-af34-4032-9c3d-d7e95b856f36/network-check-target-container/0.log"
Aug 01 00:39:37.488209 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:37.488182    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-l94lx_85746df1-2720-4d80-96a6-ed16dbee95a8/iptables-alerter/0.log"
Aug 01 00:39:38.033044 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:38.033005    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-9bmfk_0fe4a567-1748-4bd0-bc6f-244498c60b37/tuned/0.log"
Aug 01 00:39:39.585678 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:39.585653    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-559cbf5745-mfkbb_e87d9312-d1ae-4934-b93c-6c53272c30a4/cluster-samples-operator/0.log"
Aug 01 00:39:39.599777 ip-10-0-140-138 kubenswrapper[2546]: I0801 00:39:39.599757    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-559cbf5745-mfkbb_e87d9312-d1ae-4934-b93c-6c53272c30a4/cluster-samples-operator-watch/0.log"