Aug 05 17:10:27.520479 ip-10-0-142-192 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Aug 05 17:10:27.520493 ip-10-0-142-192 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Aug 05 17:10:27.520504 ip-10-0-142-192 systemd[1]: kubelet.service: Failed with result 'resources'.
Aug 05 17:10:27.520815 ip-10-0-142-192 systemd[1]: Failed to start Kubernetes Kubelet.
Aug 05 17:10:37.571814 ip-10-0-142-192 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Aug 05 17:10:37.571833 ip-10-0-142-192 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6f79d351418d4e84bcadc7f3176700ff --
Aug 05 17:12:28.208825 ip-10-0-142-192 systemd[1]: Starting Kubernetes Kubelet...
Aug 05 17:12:29.980359 ip-10-0-142-192 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 05 17:12:29.980359 ip-10-0-142-192 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 05 17:12:29.980359 ip-10-0-142-192 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 05 17:12:29.980359 ip-10-0-142-192 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 05 17:12:29.980359 ip-10-0-142-192 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 05 17:12:29.982751 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.982665    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 05 17:12:29.988730 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988713    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:29.988730 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988729    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:29.988730 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988733    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988737    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988740    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988743    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988746    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988749    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988752    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988757    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988760    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988763    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988775    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988777    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988786    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988789    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988792    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988795    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988798    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988802    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988806    2574 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988810    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:29.988846 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988813    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988818    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988821    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988826    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988830    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988833    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988836    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988838    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988842    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988844    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988847    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988850    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988853    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988858    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988862    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988865    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988868    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988871    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988874    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:29.989338 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988877    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988880    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988884    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988887    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988890    2574 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988893    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988897    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988899    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988902    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988905    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988907    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988910    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988913    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988915    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988918    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988922    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988925    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988928    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988931    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:29.989806 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988933    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988937    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988942    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988946    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988949    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988952    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988954    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988957    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988959    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988962    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988965    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988967    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988970    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988973    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988976    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988978    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988981    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988984    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988986    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988989    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:29.990295 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988991    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988995    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.988997    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989000    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989002    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989005    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989007    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989010    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989013    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989017    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989020    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989023    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989026    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989029    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989032    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989035    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989037    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989040    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:29.990778 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.989043    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991140    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991148    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991151    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991154    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991157    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991160    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991163    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991166    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991169    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991172    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991175    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991178    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991180    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991183    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991186    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991189    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991192    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991194    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:29.991227 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991197    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991200    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991202    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991205    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991208    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991210    2574 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991213    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991215    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991218    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991220    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991223    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991227    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991230    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991234    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991237    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991240    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991243    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991245    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991248    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991250    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:29.991757 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991253    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991255    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991259    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991263    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991266    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991269    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991272    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991274    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991277    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991280    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991283    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991285    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991288    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991290    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991293    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991296    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991300    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991304    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991307    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:29.992266 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991311    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991313    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991316    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991319    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991321    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991324    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991326    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991329    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991332    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991334    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991337    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991339    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991343    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991345    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991348    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991352    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991354    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991357    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991360    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:29.992762 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991362    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991365    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991367    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991370    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991374    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991377    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991379    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991382    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991386    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991389    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991392    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991394    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991397    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991399    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991402    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991405    2574 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991408    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991410    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991413    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991415    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991418    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:29.993258 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991420    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991423    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991826    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991831    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991834    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991836    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991839    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991842    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991844    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991847    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991849    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991852    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991855    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991857    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991860    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991863    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991866    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991878    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991881    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991884    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:29.993775 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991887    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991890    2574 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991893    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991896    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991899    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991901    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991904    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991907    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991909    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991912    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991915    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991918    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991921    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991926    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991929    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991932    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991934    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991936    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991939    2574 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:29.994328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991941    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991944    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991947    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991949    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991951    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991954    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991957    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991959    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991962    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991965    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991968    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991971    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991974    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991978    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991981    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991983    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991986    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991989    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991991    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991994    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:29.994788 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991996    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.991999    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992001    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992004    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992006    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992009    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992014    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992018    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992020    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992024    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992028    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992030    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992033    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992036    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992039    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992041    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992043    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992046    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992049    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:29.995328 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992051    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992054    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992056    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992059    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992062    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992065    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992067    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992082    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992085    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992088    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992090    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992093    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992096    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992098    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992101    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992104    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992106    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992109    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992111    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992114    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:29.995774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992116    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992119    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992121    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992195    2574 flags.go:64] FLAG: --address="0.0.0.0"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992203    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992208    2574 flags.go:64] FLAG: --anonymous-auth="true"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992213    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992217    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992221    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992225    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992230    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992233    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992236    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992240    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992244    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992247    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992251    2574 flags.go:64] FLAG: --cgroup-root=""
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992254    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992257    2574 flags.go:64] FLAG: --client-ca-file=""
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992260    2574 flags.go:64] FLAG: --cloud-provider="external"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992263    2574 flags.go:64] FLAG: --cluster-dns="[]"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992268    2574 flags.go:64] FLAG: --cluster-domain=""
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992271    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 05 17:12:29.996275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992275    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992278    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992282    2574 flags.go:64] FLAG: --container-log-max-files="5"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992286    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992289    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992292    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992295    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992298    2574 flags.go:64] FLAG: --contention-profiling="false"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992301    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992304    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992307    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992310    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992315    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992318    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992321    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992324    2574 flags.go:64] FLAG: --enable-load-reader="false"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992327    2574 flags.go:64] FLAG: --enable-server="true"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992329    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992334    2574 flags.go:64] FLAG: --event-burst="100"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992337    2574 flags.go:64] FLAG: --event-qps="50"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992340    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992343    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992346    2574 flags.go:64] FLAG: --eviction-hard=""
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992350    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 05 17:12:29.996839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992353    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992359    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992363    2574 flags.go:64] FLAG: --eviction-soft=""
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992366    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992369    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992372    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992375    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992378    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992381    2574 flags.go:64] FLAG: --fail-swap-on="true"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992384    2574 flags.go:64] FLAG: --feature-gates=""
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992387    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992391    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992394    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992397    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992400    2574 flags.go:64] FLAG: --healthz-port="10248"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992403    2574 flags.go:64] FLAG: --help="false"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992406    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-142-192.ec2.internal"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992409    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992412    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992415    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992419    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992422    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992425    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 05 17:12:29.997442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992428    2574 flags.go:64] FLAG: --image-service-endpoint=""
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992431    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992433    2574 flags.go:64] FLAG: --kube-api-burst="100"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992438    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992441    2574 flags.go:64] FLAG: --kube-api-qps="50"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992444    2574 flags.go:64] FLAG: --kube-reserved=""
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992447    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992450    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992453    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992456    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992462    2574 flags.go:64] FLAG: --lock-file=""
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992466    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992469    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992472    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992478    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992482    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992485    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992487    2574 flags.go:64] FLAG: --logging-format="text"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992490    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992494    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992497    2574 flags.go:64] FLAG: --manifest-url=""
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992500    2574 flags.go:64] FLAG: --manifest-url-header=""
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992504    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992507    2574 flags.go:64] FLAG: --max-open-files="1000000"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992517    2574 flags.go:64] FLAG: --max-pods="110"
Aug 05 17:12:29.998028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992520    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992523    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992526    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992529    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992532    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992535    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992538    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992545    2574 flags.go:64] FLAG: --node-status-max-images="50"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992548    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992552    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992556    2574 flags.go:64] FLAG: --pod-cidr=""
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992559    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992565    2574 flags.go:64] FLAG: --pod-manifest-path=""
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992568    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992571    2574 flags.go:64] FLAG: --pods-per-core="0"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992575    2574 flags.go:64] FLAG: --port="10250"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992578    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992581    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06f639a4efc65d458"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992586    2574 flags.go:64] FLAG: --qos-reserved=""
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992589    2574 flags.go:64] FLAG: --read-only-port="10255"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992593    2574 flags.go:64] FLAG: --register-node="true"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992596    2574 flags.go:64] FLAG: --register-with-taints=""
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992600    2574 flags.go:64] FLAG: --registry-burst="10"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992603    2574 flags.go:64] FLAG: --registry-qps="5"
Aug 05 17:12:29.998671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992606    2574 flags.go:64] FLAG: --reserved-cpus=""
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992609    2574 flags.go:64] FLAG: --reserved-memory=""
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992612    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992616    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992619    2574 flags.go:64] FLAG: --rotate-certificates="false"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992623    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992626    2574 flags.go:64] FLAG: --runonce="false"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992629    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992632    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992635    2574 flags.go:64] FLAG: --seccomp-default="false"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992638    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992641    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992644    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992647    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992651    2574 flags.go:64] FLAG: --storage-driver-password="root"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992654    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992657    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992660    2574 flags.go:64] FLAG: --storage-driver-user="root"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992665    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992668    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992671    2574 flags.go:64] FLAG: --system-cgroups=""
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992674    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992679    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992682    2574 flags.go:64] FLAG: --tls-cert-file=""
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992685    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 05 17:12:29.999655 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992689    2574 flags.go:64] FLAG: --tls-min-version=""
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992693    2574 flags.go:64] FLAG: --tls-private-key-file=""
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992696    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992699    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992703    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992706    2574 flags.go:64] FLAG: --v="2"
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992710    2574 flags.go:64] FLAG: --version="false"
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992714    2574 flags.go:64] FLAG: --vmodule=""
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992718    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.992721    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992838    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992841    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992844    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992847    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992849    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992852    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992855    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992857    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992860    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992862    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992865    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992868    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:30.000461 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992870    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992873    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992875    2574 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992879    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992882    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992885    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992887    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992890    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992892    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992895    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992898    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992903    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992906    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992909    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992912    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992915    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992917    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992920    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992923    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992925    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:30.001000 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992928    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992931    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992934    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992937    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992940    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992942    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992945    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992947    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992950    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992953    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992955    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992958    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992960    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992964    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992968    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992972    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992975    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992978    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992981    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:30.001522 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992984    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992986    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992989    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992992    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.992997    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993001    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993004    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993007    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993010    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993013    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993016    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993018    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993021    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993024    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993026    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993029    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993032    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993034    2574 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993037    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:30.001999 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993040    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993042    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993045    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993048    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993050    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993053    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993055    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993058    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993061    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993065    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993068    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993084    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993087    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993090    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993093    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993096    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993098    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993102    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993105    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993107    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:30.002480 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993110    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:30.002974 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993113    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:30.002974 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993115    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:30.002974 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993118    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:30.002974 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993121    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:30.002974 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993124    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:30.002974 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993127    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:30.002974 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993129    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:30.002974 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:29.993132    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:30.002974 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:29.993138    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 05 17:12:30.002974 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.001050    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 05 17:12:30.002974 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.001067    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 05 17:12:30.002974 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.001794    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 05 17:12:30.002974 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.001802    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 05 17:12:30.002974 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001868    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:30.002974 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001873    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001877    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001881    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001884    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001887    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001890    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001893    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001896    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001900    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001904    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001908    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001911    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001914    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001917    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001919    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001922    2574 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001925    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001928    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001931    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:30.003370 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001935    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001940    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001944    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001948    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001957    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001961    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001965    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001968    2574 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001970    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001974    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001977    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001980    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001982    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001987    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001990    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001994    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.001997    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002000    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002003    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:30.003827 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002006    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002011    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002015    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002018    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002020    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002023    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002026    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002028    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002031    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002034    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002037    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002039    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002042    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002044    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002048    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002050    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002053    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002055    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002058    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002060    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:30.004322 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002063    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002065    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002069    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002086    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002090    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002093    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002096    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002099    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002101    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002104    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002107    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002109    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002112    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002115    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002118    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002120    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002123    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002126    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002128    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002131    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:30.004814 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002135    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002138    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002141    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002143    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002146    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002148    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002151    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002154    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002156    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002159    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002161    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002164    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002166    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002169    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002173    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002176    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002178    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002181    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002184    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:30.005440 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002186    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:30.005901 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.002192    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 05 17:12:30.005901 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002312    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:30.005901 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002318    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:30.005901 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002321    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:30.005901 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002324    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:30.005901 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002327    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:30.005901 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002329    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:30.005901 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002332    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:30.005901 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002334    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:30.005901 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002337    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:30.005901 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002340    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:30.005901 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002342    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:30.005901 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002346    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:30.005901 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002349    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:30.005901 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002352    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002354    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002357    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002359    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002363    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002367    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002371    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002373    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002393    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002397    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002401    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002404    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002407    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002410    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002413    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002416    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002419    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002423    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002426    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002428    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:30.006299 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002431    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002434    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002436    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002439    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002442    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002444    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002447    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002450    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002452    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002456    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002459    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002462    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002466    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002468    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002471    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002474    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002476    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002480    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002484    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002487    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:30.006787 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002490    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002493    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002495    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002498    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002500    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002503    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002507    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002509    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002512    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002514    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002517    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002520    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002522    2574 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002525    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002527    2574 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002530    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002533    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002536    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002538    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002541    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:30.007290 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002543    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002546    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002550    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002554    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002557    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002561    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002564    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002566    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002569    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002571    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002574    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002576    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002579    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002581    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002584    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002587    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002589    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002592    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002595    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:30.007773 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002597    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:30.008253 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002600    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:30.008253 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002602    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:30.008253 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002605    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:30.008253 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002607    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:30.008253 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002610    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:30.008253 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:30.002612    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:30.008253 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.002617    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 05 17:12:30.008253 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.002741    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 05 17:12:30.008253 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.005185    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 05 17:12:30.008253 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.006432    2574 server.go:1019] "Starting client certificate rotation"
Aug 05 17:12:30.008253 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.006530    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 05 17:12:30.008253 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.006563    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 05 17:12:30.008809 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.008795    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 05 17:12:30.010177 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.010163    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 05 17:12:30.020449 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.020432    2574 log.go:25] "Validated CRI v1 runtime API"
Aug 05 17:12:30.023531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.023514    2574 log.go:25] "Validated CRI v1 image API"
Aug 05 17:12:30.024242 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.024226    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 05 17:12:30.026817 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.026799    2574 fs.go:135] Filesystem UUIDs: map[016c8abf-f9df-41b7-95fe-de26a74bc823:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ebf8bc88-50ae-4f24-b3e9-5238f1549794:/dev/nvme0n1p4]
Aug 05 17:12:30.026869 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.026817    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 05 17:12:30.032112 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.031976    2574 manager.go:216] Machine: {Timestamp:2026-08-05 17:12:30.030505044 +0000 UTC m=+0.322359869 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099476 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2921d865dd58fa83e5a350ccc214cb SystemUUID:ec2921d8-65dd-58fa-83e5-a350ccc214cb BootID:6f79d351-418d-4e84-bcad-c7f3176700ff Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1a:cc:80:6d:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1a:cc:80:6d:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:58:30:5e:c6:e4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 05 17:12:30.032112 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.032102    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 05 17:12:30.032239 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.032186    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 05 17:12:30.035665 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.035630    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 05 17:12:30.035830 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.035673    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-192.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 05 17:12:30.036884 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.036871    2574 topology_manager.go:138] "Creating topology manager with none policy"
Aug 05 17:12:30.036918 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.036887    2574 container_manager_linux.go:309] "Creating device plugin manager"
Aug 05 17:12:30.036918 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.036896    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 05 17:12:30.037568 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.037556    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 05 17:12:30.037675 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.037658    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 05 17:12:30.038348 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.038337    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 05 17:12:30.041243 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.041231    2574 state_mem.go:36] "Initialized new in-memory state store"
Aug 05 17:12:30.041413 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.041404    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 05 17:12:30.041442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.041436    2574 kubelet.go:486] "Attempting to sync node with API server"
Aug 05 17:12:30.041476 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.041448    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 05 17:12:30.041476 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.041459    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 05 17:12:30.041476 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.041468    2574 kubelet.go:398] "Adding apiserver pod source"
Aug 05 17:12:30.041476 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.041477    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 05 17:12:30.041881 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.041870    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 05 17:12:30.041928 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.041889    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 05 17:12:30.044084 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.044054    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 05 17:12:30.045891 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.045877    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 05 17:12:30.045942 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.045908    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 05 17:12:30.047400 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.047387    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 05 17:12:30.047446 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.047404    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 05 17:12:30.047446 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.047411    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 05 17:12:30.047446 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.047416    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 05 17:12:30.047446 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.047425    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 05 17:12:30.047446 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.047430    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 05 17:12:30.047446 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.047438    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 05 17:12:30.047446 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.047443    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 05 17:12:30.047446 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.047449    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 05 17:12:30.047656 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.047455    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 05 17:12:30.047656 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.047463    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 05 17:12:30.047656 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.047468    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 05 17:12:30.048218 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.048206    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 05 17:12:30.048252 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.048221    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 05 17:12:30.051531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.051510    2574 server.go:1268] "Started kubelet"
Aug 05 17:12:30.051652 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.051593    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 05 17:12:30.051713 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.051671    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 05 17:12:30.051790 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.051763    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 05 17:12:30.052391 ip-10-0-142-192 systemd[1]: Started Kubernetes Kubelet.
Aug 05 17:12:30.053050 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.052954    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 05 17:12:30.053447 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.053253    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-192.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 05 17:12:30.053447 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.053333    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-192.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 05 17:12:30.053447 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.053356    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 05 17:12:30.053947 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.053920    2574 server.go:310] "Adding debug handlers to kubelet server"
Aug 05 17:12:30.056668 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.056634    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 05 17:12:30.057675 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.057655    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 05 17:12:30.057675 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.057664    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 05 17:12:30.057836 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.057793    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-192.ec2.internal\" not found"
Aug 05 17:12:30.058241 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.058224    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 05 17:12:30.058304 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.058225    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 05 17:12:30.058304 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.058257    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 05 17:12:30.058404 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.058378    2574 reconstruct.go:97] "Volume reconstruction finished"
Aug 05 17:12:30.058404 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.058387    2574 reconciler.go:29] "Reconciler: start to sync state"
Aug 05 17:12:30.058785 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.058768    2574 factory.go:153] Registering CRI-O factory
Aug 05 17:12:30.058870 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.058788    2574 factory.go:223] Registration of the crio container factory successfully
Aug 05 17:12:30.058870 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.058844    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 05 17:12:30.058870 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.058854    2574 factory.go:55] Registering systemd factory
Aug 05 17:12:30.058870 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.058862    2574 factory.go:223] Registration of the systemd container factory successfully
Aug 05 17:12:30.059160 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.058885    2574 factory.go:103] Registering Raw factory
Aug 05 17:12:30.059160 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.058900    2574 manager.go:1196] Started watching for new ooms in manager
Aug 05 17:12:30.059557 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.059542    2574 manager.go:349] Starting recovery of all containers
Aug 05 17:12:30.060428 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.060401    2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-192.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Aug 05 17:12:30.061213 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.059890    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-192.ec2.internal.18c8f8046771f2c2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-192.ec2.internal,UID:ip-10-0-142-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-192.ec2.internal,},FirstTimestamp:2026-08-05 17:12:30.051480258 +0000 UTC m=+0.343335081,LastTimestamp:2026-08-05 17:12:30.051480258 +0000 UTC m=+0.343335081,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-192.ec2.internal,}"
Aug 05 17:12:30.066063 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.066031    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Aug 05 17:12:30.071450 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.071426    2574 manager.go:354] Recovery completed
Aug 05 17:12:30.075755 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.075654    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:30.077578 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.077561    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:30.077665 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.077597    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:30.077665 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.077612    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:30.078184 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.078171    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 05 17:12:30.078184 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.078180    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:30.078282 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.078183    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 05 17:12:30.078282 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.078218    2574 state_mem.go:36] "Initialized new in-memory state store"
Aug 05 17:12:30.078991 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.078977    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:30.079031 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.079003    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:30.079031 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.079014    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:30.079428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.079417    2574 policy_none.go:49] "None policy: Start"
Aug 05 17:12:30.079459 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.079432    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 05 17:12:30.079459 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.079441    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 05 17:12:30.080324 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.080253    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-192.ec2.internal.18c8f80469002eb8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-192.ec2.internal,UID:ip-10-0-142-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-192.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-192.ec2.internal,},FirstTimestamp:2026-08-05 17:12:30.077578936 +0000 UTC m=+0.369433764,LastTimestamp:2026-08-05 17:12:30.077578936 +0000 UTC m=+0.369433764,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-192.ec2.internal,}"
Aug 05 17:12:30.080425 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.080411    2574 policy_none.go:47] "Start"
Aug 05 17:12:30.090262 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.090194    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-192.ec2.internal.18c8f80469009205  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-192.ec2.internal,UID:ip-10-0-142-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-192.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-192.ec2.internal,},FirstTimestamp:2026-08-05 17:12:30.077604357 +0000 UTC m=+0.369459187,LastTimestamp:2026-08-05 17:12:30.077604357 +0000 UTC m=+0.369459187,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-192.ec2.internal,}"
Aug 05 17:12:30.099774 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.099739    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ms7ff"
Aug 05 17:12:30.102229 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.102157    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-192.ec2.internal.18c8f8046900c7f1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-192.ec2.internal,UID:ip-10-0-142-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-192.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-192.ec2.internal,},FirstTimestamp:2026-08-05 17:12:30.077618161 +0000 UTC m=+0.369472987,LastTimestamp:2026-08-05 17:12:30.077618161 +0000 UTC m=+0.369472987,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-192.ec2.internal,}"
Aug 05 17:12:30.109749 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.109728    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ms7ff"
Aug 05 17:12:30.112247 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.112173    2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-192.ec2.internal.18c8f80469002eb8\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-192.ec2.internal.18c8f80469002eb8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-192.ec2.internal,UID:ip-10-0-142-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-192.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-192.ec2.internal,},FirstTimestamp:2026-08-05 17:12:30.077578936 +0000 UTC m=+0.369433764,LastTimestamp:2026-08-05 17:12:30.078992341 +0000 UTC m=+0.370847163,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-192.ec2.internal,}"
Aug 05 17:12:30.115743 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.115728    2574 manager.go:337] "Starting Device Plugin manager"
Aug 05 17:12:30.115832 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.115760    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 05 17:12:30.115832 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.115770    2574 server.go:87] "Starting device plugin registration server"
Aug 05 17:12:30.116046 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.116036    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 05 17:12:30.116114 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.116051    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 05 17:12:30.116424 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.116232    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 05 17:12:30.116424 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.116339    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 05 17:12:30.116424 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.116358    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 05 17:12:30.116995 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.116976    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 05 17:12:30.117069 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.117032    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-192.ec2.internal\" not found"
Aug 05 17:12:30.187205 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.187159    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 05 17:12:30.188380 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.188351    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 05 17:12:30.188437 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.188390    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 05 17:12:30.188437 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.188414    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 05 17:12:30.188527 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.188451    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 05 17:12:30.190706 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.190683    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:30.216847 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.216818    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:30.217765 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.217747    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:30.217853 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.217780    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:30.217853 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.217790    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:30.217853 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.217813    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.225227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.225178    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.225227 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.225200    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-192.ec2.internal\": node \"ip-10-0-142-192.ec2.internal\" not found"
Aug 05 17:12:30.240207 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.240186    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-192.ec2.internal\" not found"
Aug 05 17:12:30.288861 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.288806    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-192.ec2.internal"]
Aug 05 17:12:30.289036 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.288903    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:30.289890 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.289873    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:30.289980 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.289909    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:30.289980 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.289923    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:30.291413 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.291396    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:30.291531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.291503    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.291577 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.291533    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:30.292475 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.292459    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:30.292567 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.292482    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:30.292567 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.292492    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:30.292567 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.292506    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:30.292567 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.292506    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:30.292690 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.292573    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:30.293688 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.293672    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.293784 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.293695    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:30.294374 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.294357    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:30.294466 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.294388    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:30.294466 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.294434    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:30.315799 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.315775    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-192.ec2.internal\" not found" node="ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.320188 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.320171    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-192.ec2.internal\" not found" node="ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.341248 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.341222    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-192.ec2.internal\" not found"
Aug 05 17:12:30.360510 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.360487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/322b4eab99b8f4f207438c9174b617a8-config\") pod \"kube-apiserver-proxy-ip-10-0-142-192.ec2.internal\" (UID: \"322b4eab99b8f4f207438c9174b617a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.360596 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.360514    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36f6ab0e8ee2c59be4633f740c4f4bb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal\" (UID: \"36f6ab0e8ee2c59be4633f740c4f4bb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.360596 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.360538    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36f6ab0e8ee2c59be4633f740c4f4bb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal\" (UID: \"36f6ab0e8ee2c59be4633f740c4f4bb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.442149 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.442110    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-192.ec2.internal\" not found"
Aug 05 17:12:30.461513 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.461479    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/322b4eab99b8f4f207438c9174b617a8-config\") pod \"kube-apiserver-proxy-ip-10-0-142-192.ec2.internal\" (UID: \"322b4eab99b8f4f207438c9174b617a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.461513 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.461511    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36f6ab0e8ee2c59be4633f740c4f4bb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal\" (UID: \"36f6ab0e8ee2c59be4633f740c4f4bb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.461691 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.461533    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36f6ab0e8ee2c59be4633f740c4f4bb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal\" (UID: \"36f6ab0e8ee2c59be4633f740c4f4bb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.461691 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.461586    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36f6ab0e8ee2c59be4633f740c4f4bb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal\" (UID: \"36f6ab0e8ee2c59be4633f740c4f4bb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.461691 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.461590    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/322b4eab99b8f4f207438c9174b617a8-config\") pod \"kube-apiserver-proxy-ip-10-0-142-192.ec2.internal\" (UID: \"322b4eab99b8f4f207438c9174b617a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.461691 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.461593    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36f6ab0e8ee2c59be4633f740c4f4bb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal\" (UID: \"36f6ab0e8ee2c59be4633f740c4f4bb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.542873 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.542779    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-192.ec2.internal\" not found"
Aug 05 17:12:30.618291 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.618257    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.622482 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:30.622449    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-192.ec2.internal"
Aug 05 17:12:30.643868 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.643833    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-192.ec2.internal\" not found"
Aug 05 17:12:30.744387 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.744351    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-192.ec2.internal\" not found"
Aug 05 17:12:30.844933 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.844860    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-192.ec2.internal\" not found"
Aug 05 17:12:30.945505 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:30.945463    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-192.ec2.internal\" not found"
Aug 05 17:12:31.002535 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.002506    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:31.005703 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.005674    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 05 17:12:31.005814 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.005797    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 05 17:12:31.005850 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.005831    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 05 17:12:31.042010 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.041978    2574 apiserver.go:52] "Watching apiserver"
Aug 05 17:12:31.058502 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.058479    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 05 17:12:31.058502 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.058489    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal"
Aug 05 17:12:31.060971 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.060951    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 05 17:12:31.061310 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.061291    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-b78nx","openshift-dns/node-resolver-kgdd2","openshift-image-registry/node-ca-hsn68","openshift-multus/network-metrics-daemon-xwnbn","openshift-network-diagnostics/network-check-target-44mzx","openshift-network-operator/iptables-alerter-ldtgp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd","openshift-multus/multus-2slm7","openshift-multus/multus-additional-cni-plugins-n6mhh","openshift-ovn-kubernetes/ovnkube-node-xrhbh","kube-system/global-pull-secret-syncer-w89xx","kube-system/konnectivity-agent-wk6ks"]
Aug 05 17:12:31.062953 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.062932    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.064137 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.064115    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kgdd2"
Aug 05 17:12:31.065221 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.065201    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hsn68"
Aug 05 17:12:31.065867 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.065853    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f4f2n\""
Aug 05 17:12:31.066437 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.066422    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 05 17:12:31.066567 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.066552    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 05 17:12:31.066627 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.066542    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kpwd7\""
Aug 05 17:12:31.066627 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.066620    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:31.066720 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.066619    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 05 17:12:31.066720 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.066683    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:12:31.066815 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.066800    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 05 17:12:31.067413 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.067396    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 05 17:12:31.067541 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.067526    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:31.067622 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.067603    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:12:31.067708 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.067695    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-flk8w\""
Aug 05 17:12:31.068363 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.068350    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 05 17:12:31.068427 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.068381    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 05 17:12:31.068777 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.068764    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ldtgp"
Aug 05 17:12:31.069887 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.069871    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.070898 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.070877    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.072047 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.072029    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.074663 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.074641    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 05 17:12:31.074861 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.074838    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.074924 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.074907    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-84msk\""
Aug 05 17:12:31.075180 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.075147    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 05 17:12:31.075248 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.075169    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 05 17:12:31.075248 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.075230    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 05 17:12:31.075349 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.075329    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hphct\""
Aug 05 17:12:31.075405 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.075353    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 05 17:12:31.075476 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.075457    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 05 17:12:31.075696 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.075674    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 05 17:12:31.076039 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.076020    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 05 17:12:31.076158 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.076088    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mwcfr\""
Aug 05 17:12:31.076158 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.076167    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 05 17:12:31.076387 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.076364    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 05 17:12:31.076513 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.076441    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:31.076588 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.076542    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:31.076588 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.076580    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 05 17:12:31.076901 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.076633    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 05 17:12:31.076901 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.076873    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cq9tv\""
Aug 05 17:12:31.077939 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.077726    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 05 17:12:31.077939 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.077813    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-192.ec2.internal"
Aug 05 17:12:31.077939 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.077839    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 05 17:12:31.078111 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.078012    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 05 17:12:31.078292 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.078278    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 05 17:12:31.078367 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.078355    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 05 17:12:31.078815 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.078803    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r6cl4\""
Aug 05 17:12:31.078856 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.078828    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal"]
Aug 05 17:12:31.078918 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.078905    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wk6ks"
Aug 05 17:12:31.081342 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.081325    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 05 17:12:31.082222 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.082208    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 05 17:12:31.082613 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.082599    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 05 17:12:31.082689 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.082630    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 05 17:12:31.082689 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.082651    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 05 17:12:31.082689 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.082630    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-56jxq\""
Aug 05 17:12:31.094067 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.094046    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 05 17:12:31.094357 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.094341    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-192.ec2.internal"]
Aug 05 17:12:31.114513 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.114461    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-04 17:07:30 +0000 UTC" deadline="2028-03-09 02:37:27.669832293 +0000 UTC"
Aug 05 17:12:31.114513 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.114510    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13953h24m56.555325742s"
Aug 05 17:12:31.122414 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.122386    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gg72q"
Aug 05 17:12:31.132302 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.132273    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gg72q"
Aug 05 17:12:31.159204 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.159180    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 05 17:12:31.164755 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.164736    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-kubernetes\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.164852 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.164764    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77260dfe-e37a-487f-8855-324bbed53c7a-etc-tuned\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.164852 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.164785    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnrtp\" (UniqueName: \"kubernetes.io/projected/ef418180-36b0-450a-8ac5-8518f605f717-kube-api-access-jnrtp\") pod \"node-ca-hsn68\" (UID: \"ef418180-36b0-450a-8ac5-8518f605f717\") " pod="openshift-image-registry/node-ca-hsn68"
Aug 05 17:12:31.164852 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.164813    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5286a5af-946f-4116-8ec5-33c3d9dbafb7-iptables-alerter-script\") pod \"iptables-alerter-ldtgp\" (UID: \"5286a5af-946f-4116-8ec5-33c3d9dbafb7\") " pod="openshift-network-operator/iptables-alerter-ldtgp"
Aug 05 17:12:31.164852 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.164837    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-var-lib-cni-multus\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.165000 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.164855    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.165000 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.164873    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/03b7f246-02d1-4207-94ff-1264fb006a98-ovnkube-config\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.165000 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.164891    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-run-multus-certs\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.165000 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.164914    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-slash\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.165000 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.164932    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nn8v\" (UniqueName: \"kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v\") pod \"network-check-target-44mzx\" (UID: \"1fc45fe8-5a2b-4e42-bf7e-938c50805465\") " pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:31.165000 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.164947    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-registration-dir\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.165000 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.164961    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-device-dir\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.165000 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.164994    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a578ff9-e0d4-4386-a305-bcd010078bab-cnibin\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.165244 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a578ff9-e0d4-4386-a305-bcd010078bab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.165244 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165093    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2347bfcd-6da3-456d-a788-a18eaf3dbb61-cni-binary-copy\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.165244 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165120    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-cni-netd\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.165244 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165158    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-socket-dir\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.165244 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165176    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4a578ff9-e0d4-4386-a305-bcd010078bab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.165244 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165192    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-sysconfig\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.165244 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165206    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ef418180-36b0-450a-8ac5-8518f605f717-serviceca\") pod \"node-ca-hsn68\" (UID: \"ef418180-36b0-450a-8ac5-8518f605f717\") " pod="openshift-image-registry/node-ca-hsn68"
Aug 05 17:12:31.165244 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165229    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-multus-cni-dir\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.165500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165245    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-run-netns\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.165500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165261    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlljz\" (UniqueName: \"kubernetes.io/projected/03b7f246-02d1-4207-94ff-1264fb006a98-kube-api-access-hlljz\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.165500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165277    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-sys-fs\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.165500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165291    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4a578ff9-e0d4-4386-a305-bcd010078bab-os-release\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.165500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165312    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-sysctl-conf\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.165500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165354    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-lib-modules\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.165500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165374    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-node-log\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.165500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165431    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-run-ovn-kubernetes\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.165500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165448    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/03b7f246-02d1-4207-94ff-1264fb006a98-ovnkube-script-lib\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.165500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165464    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-host\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.165500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-run-k8s-cni-cncf-io\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165505    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-run-systemd\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165520    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a578ff9-e0d4-4386-a305-bcd010078bab-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165564    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4a578ff9-e0d4-4386-a305-bcd010078bab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165594    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef418180-36b0-450a-8ac5-8518f605f717-host\") pod \"node-ca-hsn68\" (UID: \"ef418180-36b0-450a-8ac5-8518f605f717\") " pod="openshift-image-registry/node-ca-hsn68"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-os-release\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165643    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-var-lib-kubelet\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165666    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f1ee110b-ab37-4c60-b72c-74f07b55668d-agent-certs\") pod \"konnectivity-agent-wk6ks\" (UID: \"f1ee110b-ab37-4c60-b72c-74f07b55668d\") " pod="kube-system/konnectivity-agent-wk6ks"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165690    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-var-lib-kubelet\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-var-lib-cni-bin\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165720    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2347bfcd-6da3-456d-a788-a18eaf3dbb61-multus-daemon-config\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165740    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-log-socket\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165762    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-cni-bin\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165777    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-systemd-units\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165797    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbzbl\" (UniqueName: \"kubernetes.io/projected/fc979956-0bb5-4600-81bb-19a5521be56d-kube-api-access-tbzbl\") pod \"node-resolver-kgdd2\" (UID: \"fc979956-0bb5-4600-81bb-19a5521be56d\") " pod="openshift-dns/node-resolver-kgdd2"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165827    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hkrg\" (UniqueName: \"kubernetes.io/projected/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-kube-api-access-9hkrg\") pod \"network-metrics-daemon-xwnbn\" (UID: \"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660\") " pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:31.165937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165912    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-systemd\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165933    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-sys\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165966    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-system-cni-dir\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.165984    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-multus-conf-dir\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166003    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-run-openvswitch\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166034    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee110b-ab37-4c60-b72c-74f07b55668d-konnectivity-ca\") pod \"konnectivity-agent-wk6ks\" (UID: \"f1ee110b-ab37-4c60-b72c-74f07b55668d\") " pod="kube-system/konnectivity-agent-wk6ks"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166082    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-modprobe-d\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166113    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5286a5af-946f-4116-8ec5-33c3d9dbafb7-host-slash\") pod \"iptables-alerter-ldtgp\" (UID: \"5286a5af-946f-4116-8ec5-33c3d9dbafb7\") " pod="openshift-network-operator/iptables-alerter-ldtgp"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166133    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtwhr\" (UniqueName: \"kubernetes.io/projected/5286a5af-946f-4116-8ec5-33c3d9dbafb7-kube-api-access-gtwhr\") pod \"iptables-alerter-ldtgp\" (UID: \"5286a5af-946f-4116-8ec5-33c3d9dbafb7\") " pod="openshift-network-operator/iptables-alerter-ldtgp"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166152    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-cnibin\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/03b7f246-02d1-4207-94ff-1264fb006a98-ovn-node-metrics-cert\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166191    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-run\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166204    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77260dfe-e37a-487f-8855-324bbed53c7a-tmp\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166226    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-kubelet\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166251    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs\") pod \"network-metrics-daemon-xwnbn\" (UID: \"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660\") " pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:31.166428 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166271    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grp48\" (UniqueName: \"kubernetes.io/projected/6e6fb427-9f41-42ec-9c75-c9c49caebf28-kube-api-access-grp48\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166292    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a578ff9-e0d4-4386-a305-bcd010078bab-system-cni-dir\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166312    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-kubelet-config\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166339    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-sysctl-d\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166357    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-etc-openvswitch\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166371    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-run-ovn\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166398    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-dbus\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166436    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-etc-kubernetes\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166455    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c58t9\" (UniqueName: \"kubernetes.io/projected/2347bfcd-6da3-456d-a788-a18eaf3dbb61-kube-api-access-c58t9\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166470    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtlgs\" (UniqueName: \"kubernetes.io/projected/77260dfe-e37a-487f-8855-324bbed53c7a-kube-api-access-mtlgs\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166492    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-hostroot\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166513    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc979956-0bb5-4600-81bb-19a5521be56d-tmp-dir\") pod \"node-resolver-kgdd2\" (UID: \"fc979956-0bb5-4600-81bb-19a5521be56d\") " pod="openshift-dns/node-resolver-kgdd2"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166528    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166542    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-multus-socket-dir-parent\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166554    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-run-netns\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166576    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-var-lib-openvswitch\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166594    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/03b7f246-02d1-4207-94ff-1264fb006a98-env-overrides\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.166994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166634    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc979956-0bb5-4600-81bb-19a5521be56d-hosts-file\") pod \"node-resolver-kgdd2\" (UID: \"fc979956-0bb5-4600-81bb-19a5521be56d\") " pod="openshift-dns/node-resolver-kgdd2"
Aug 05 17:12:31.167481 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-etc-selinux\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.167481 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.166701    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swj8q\" (UniqueName: \"kubernetes.io/projected/4a578ff9-e0d4-4386-a305-bcd010078bab-kube-api-access-swj8q\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.236014 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:31.235978    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322b4eab99b8f4f207438c9174b617a8.slice/crio-3a0df816195495cd310f4c7e40b8f98f31ef1d9a92323d64ee4af9d18b669157 WatchSource:0}: Error finding container 3a0df816195495cd310f4c7e40b8f98f31ef1d9a92323d64ee4af9d18b669157: Status 404 returned error can't find the container with id 3a0df816195495cd310f4c7e40b8f98f31ef1d9a92323d64ee4af9d18b669157
Aug 05 17:12:31.236284 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:31.236269    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f6ab0e8ee2c59be4633f740c4f4bb8.slice/crio-0e8f7b24665ff901ee85b98391c79bc1701b41d65f0ba7261c717f8f1aff1cd5 WatchSource:0}: Error finding container 0e8f7b24665ff901ee85b98391c79bc1701b41d65f0ba7261c717f8f1aff1cd5: Status 404 returned error can't find the container with id 0e8f7b24665ff901ee85b98391c79bc1701b41d65f0ba7261c717f8f1aff1cd5
Aug 05 17:12:31.240861 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.240826    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 05 17:12:31.266957 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.266926    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef418180-36b0-450a-8ac5-8518f605f717-host\") pod \"node-ca-hsn68\" (UID: \"ef418180-36b0-450a-8ac5-8518f605f717\") " pod="openshift-image-registry/node-ca-hsn68"
Aug 05 17:12:31.266957 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.266953    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-os-release\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.267161 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.266968    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-var-lib-kubelet\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.267161 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.266984    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f1ee110b-ab37-4c60-b72c-74f07b55668d-agent-certs\") pod \"konnectivity-agent-wk6ks\" (UID: \"f1ee110b-ab37-4c60-b72c-74f07b55668d\") " pod="kube-system/konnectivity-agent-wk6ks"
Aug 05 17:12:31.267161 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267049    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-var-lib-kubelet\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.267161 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267059    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-os-release\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.267161 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267083    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef418180-36b0-450a-8ac5-8518f605f717-host\") pod \"node-ca-hsn68\" (UID: \"ef418180-36b0-450a-8ac5-8518f605f717\") " pod="openshift-image-registry/node-ca-hsn68"
Aug 05 17:12:31.267161 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267087    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-var-lib-kubelet\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.267161 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267137    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-var-lib-cni-bin\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.267161 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267143    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-var-lib-kubelet\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.267161 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267162    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2347bfcd-6da3-456d-a788-a18eaf3dbb61-multus-daemon-config\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-log-socket\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-cni-bin\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267215    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-systemd-units\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267236    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbzbl\" (UniqueName: \"kubernetes.io/projected/fc979956-0bb5-4600-81bb-19a5521be56d-kube-api-access-tbzbl\") pod \"node-resolver-kgdd2\" (UID: \"fc979956-0bb5-4600-81bb-19a5521be56d\") " pod="openshift-dns/node-resolver-kgdd2"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267254    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-log-socket\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267263    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hkrg\" (UniqueName: \"kubernetes.io/projected/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-kube-api-access-9hkrg\") pod \"network-metrics-daemon-xwnbn\" (UID: \"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660\") " pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267197    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-var-lib-cni-bin\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267270    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-systemd-units\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267298    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-cni-bin\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267327    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267351    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-systemd\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267366    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267385    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267374    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-sys\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267411    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-sys\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267430    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-system-cni-dir\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267454    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-multus-conf-dir\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.267500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-run-openvswitch\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267481    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-system-cni-dir\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267487    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee110b-ab37-4c60-b72c-74f07b55668d-konnectivity-ca\") pod \"konnectivity-agent-wk6ks\" (UID: \"f1ee110b-ab37-4c60-b72c-74f07b55668d\") " pod="kube-system/konnectivity-agent-wk6ks"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267429    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-systemd\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267512    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-modprobe-d\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267524    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-multus-conf-dir\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267535    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5286a5af-946f-4116-8ec5-33c3d9dbafb7-host-slash\") pod \"iptables-alerter-ldtgp\" (UID: \"5286a5af-946f-4116-8ec5-33c3d9dbafb7\") " pod="openshift-network-operator/iptables-alerter-ldtgp"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267541    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-run-openvswitch\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267576    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5286a5af-946f-4116-8ec5-33c3d9dbafb7-host-slash\") pod \"iptables-alerter-ldtgp\" (UID: \"5286a5af-946f-4116-8ec5-33c3d9dbafb7\") " pod="openshift-network-operator/iptables-alerter-ldtgp"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267590    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtwhr\" (UniqueName: \"kubernetes.io/projected/5286a5af-946f-4116-8ec5-33c3d9dbafb7-kube-api-access-gtwhr\") pod \"iptables-alerter-ldtgp\" (UID: \"5286a5af-946f-4116-8ec5-33c3d9dbafb7\") " pod="openshift-network-operator/iptables-alerter-ldtgp"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-cnibin\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/03b7f246-02d1-4207-94ff-1264fb006a98-ovn-node-metrics-cert\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267620    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-modprobe-d\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267663    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-run\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267673    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-cnibin\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267695    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77260dfe-e37a-487f-8855-324bbed53c7a-tmp\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267703    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-run\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267720    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-kubelet\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.268227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267759    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs\") pod \"network-metrics-daemon-xwnbn\" (UID: \"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660\") " pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267792    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grp48\" (UniqueName: \"kubernetes.io/projected/6e6fb427-9f41-42ec-9c75-c9c49caebf28-kube-api-access-grp48\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a578ff9-e0d4-4386-a305-bcd010078bab-system-cni-dir\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267825    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-kubelet\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267845    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-kubelet-config\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267870    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-sysctl-d\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267905    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2347bfcd-6da3-456d-a788-a18eaf3dbb61-multus-daemon-config\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267913    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-etc-openvswitch\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267941    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-run-ovn\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267968    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-dbus\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267989    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-kubelet-config\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.267992    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-etc-kubernetes\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268029    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-etc-kubernetes\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268035    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c58t9\" (UniqueName: \"kubernetes.io/projected/2347bfcd-6da3-456d-a788-a18eaf3dbb61-kube-api-access-c58t9\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268065    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee110b-ab37-4c60-b72c-74f07b55668d-konnectivity-ca\") pod \"konnectivity-agent-wk6ks\" (UID: \"f1ee110b-ab37-4c60-b72c-74f07b55668d\") " pod="kube-system/konnectivity-agent-wk6ks"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268093    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-run-ovn\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268101    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtlgs\" (UniqueName: \"kubernetes.io/projected/77260dfe-e37a-487f-8855-324bbed53c7a-kube-api-access-mtlgs\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268125    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-hostroot\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.269070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268152    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc979956-0bb5-4600-81bb-19a5521be56d-tmp-dir\") pod \"node-resolver-kgdd2\" (UID: \"fc979956-0bb5-4600-81bb-19a5521be56d\") " pod="openshift-dns/node-resolver-kgdd2"
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268157    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-hostroot\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268162    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-sysctl-d\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268168    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-etc-openvswitch\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268177    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268218    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a578ff9-e0d4-4386-a305-bcd010078bab-system-cni-dir\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268257    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-dbus\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.268293    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268325    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-multus-socket-dir-parent\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268346    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-run-netns\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.268394    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268433    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-var-lib-openvswitch\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268457    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-var-lib-openvswitch\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268476    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/03b7f246-02d1-4207-94ff-1264fb006a98-env-overrides\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268512    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-multus-socket-dir-parent\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.268561    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs podName:a7531d8b-5f4c-43e0-bc2f-d9e2a415f660 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:31.768536682 +0000 UTC m=+2.060391514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs") pod "network-metrics-daemon-xwnbn" (UID: "a7531d8b-5f4c-43e0-bc2f-d9e2a415f660") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268574    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc979956-0bb5-4600-81bb-19a5521be56d-tmp-dir\") pod \"node-resolver-kgdd2\" (UID: \"fc979956-0bb5-4600-81bb-19a5521be56d\") " pod="openshift-dns/node-resolver-kgdd2"
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268586    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc979956-0bb5-4600-81bb-19a5521be56d-hosts-file\") pod \"node-resolver-kgdd2\" (UID: \"fc979956-0bb5-4600-81bb-19a5521be56d\") " pod="openshift-dns/node-resolver-kgdd2"
Aug 05 17:12:31.269985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268636    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-etc-selinux\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.268645    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret podName:057a0fc7-bbc2-47e0-9b9f-d4e292b354f1 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:31.768628838 +0000 UTC m=+2.060483663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret") pod "global-pull-secret-syncer-w89xx" (UID: "057a0fc7-bbc2-47e0-9b9f-d4e292b354f1") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268684    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swj8q\" (UniqueName: \"kubernetes.io/projected/4a578ff9-e0d4-4386-a305-bcd010078bab-kube-api-access-swj8q\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268713    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-kubernetes\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268737    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-etc-selinux\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268750    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77260dfe-e37a-487f-8855-324bbed53c7a-etc-tuned\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268772    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnrtp\" (UniqueName: \"kubernetes.io/projected/ef418180-36b0-450a-8ac5-8518f605f717-kube-api-access-jnrtp\") pod \"node-ca-hsn68\" (UID: \"ef418180-36b0-450a-8ac5-8518f605f717\") " pod="openshift-image-registry/node-ca-hsn68"
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268782    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc979956-0bb5-4600-81bb-19a5521be56d-hosts-file\") pod \"node-resolver-kgdd2\" (UID: \"fc979956-0bb5-4600-81bb-19a5521be56d\") " pod="openshift-dns/node-resolver-kgdd2"
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268793    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5286a5af-946f-4116-8ec5-33c3d9dbafb7-iptables-alerter-script\") pod \"iptables-alerter-ldtgp\" (UID: \"5286a5af-946f-4116-8ec5-33c3d9dbafb7\") " pod="openshift-network-operator/iptables-alerter-ldtgp"
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268831    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-var-lib-cni-multus\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268860    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/03b7f246-02d1-4207-94ff-1264fb006a98-ovnkube-config\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-run-multus-certs\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268962    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/03b7f246-02d1-4207-94ff-1264fb006a98-env-overrides\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.268977    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-slash\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269003    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nn8v\" (UniqueName: \"kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v\") pod \"network-check-target-44mzx\" (UID: \"1fc45fe8-5a2b-4e42-bf7e-938c50805465\") " pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:31.270863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269019    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-var-lib-cni-multus\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269028    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-registration-dir\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269036    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-kubernetes\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269095    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-slash\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269300    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5286a5af-946f-4116-8ec5-33c3d9dbafb7-iptables-alerter-script\") pod \"iptables-alerter-ldtgp\" (UID: \"5286a5af-946f-4116-8ec5-33c3d9dbafb7\") " pod="openshift-network-operator/iptables-alerter-ldtgp"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269305    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-run-multus-certs\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269305    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-registration-dir\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269335    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-device-dir\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269361    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-run-netns\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269419    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a578ff9-e0d4-4386-a305-bcd010078bab-cnibin\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269443    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-device-dir\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269462    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/03b7f246-02d1-4207-94ff-1264fb006a98-ovnkube-config\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269504    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a578ff9-e0d4-4386-a305-bcd010078bab-cnibin\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269516    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a578ff9-e0d4-4386-a305-bcd010078bab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269542    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2347bfcd-6da3-456d-a788-a18eaf3dbb61-cni-binary-copy\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269644    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-cni-netd\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.271642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269655    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a578ff9-e0d4-4386-a305-bcd010078bab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269672    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-socket-dir\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269710    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-cni-netd\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269740    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4a578ff9-e0d4-4386-a305-bcd010078bab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269768    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-sysconfig\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269787    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-socket-dir\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269808    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ef418180-36b0-450a-8ac5-8518f605f717-serviceca\") pod \"node-ca-hsn68\" (UID: \"ef418180-36b0-450a-8ac5-8518f605f717\") " pod="openshift-image-registry/node-ca-hsn68"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269832    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-multus-cni-dir\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269859    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-run-netns\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269883    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlljz\" (UniqueName: \"kubernetes.io/projected/03b7f246-02d1-4207-94ff-1264fb006a98-kube-api-access-hlljz\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269906    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-sys-fs\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269930    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4a578ff9-e0d4-4386-a305-bcd010078bab-os-release\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269957    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-sysctl-conf\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269987    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-lib-modules\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.269986    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2347bfcd-6da3-456d-a788-a18eaf3dbb61-cni-binary-copy\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270011    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-node-log\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270011    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-run-netns\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.272147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270050    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-run-ovn-kubernetes\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270067    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-multus-cni-dir\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270096    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-sysconfig\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270110    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/03b7f246-02d1-4207-94ff-1264fb006a98-ovnkube-script-lib\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270138    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-host\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270161    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-run-k8s-cni-cncf-io\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270184    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-run-systemd\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270207    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a578ff9-e0d4-4386-a305-bcd010078bab-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270210    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-lib-modules\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270242    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4a578ff9-e0d4-4386-a305-bcd010078bab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270257    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-host-run-ovn-kubernetes\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270326    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4a578ff9-e0d4-4386-a305-bcd010078bab-os-release\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270330    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e6fb427-9f41-42ec-9c75-c9c49caebf28-sys-fs\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270323    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4a578ff9-e0d4-4386-a305-bcd010078bab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270333    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-node-log\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270384    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2347bfcd-6da3-456d-a788-a18eaf3dbb61-host-run-k8s-cni-cncf-io\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270392    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-host\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.272616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270422    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77260dfe-e37a-487f-8855-324bbed53c7a-etc-sysctl-conf\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.273128 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270494    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/03b7f246-02d1-4207-94ff-1264fb006a98-run-systemd\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.273128 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270852    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a578ff9-e0d4-4386-a305-bcd010078bab-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.273128 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270869    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ef418180-36b0-450a-8ac5-8518f605f717-serviceca\") pod \"node-ca-hsn68\" (UID: \"ef418180-36b0-450a-8ac5-8518f605f717\") " pod="openshift-image-registry/node-ca-hsn68"
Aug 05 17:12:31.273128 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270878    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/03b7f246-02d1-4207-94ff-1264fb006a98-ovnkube-script-lib\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.273128 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.270905    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77260dfe-e37a-487f-8855-324bbed53c7a-tmp\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.273128 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.271018    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4a578ff9-e0d4-4386-a305-bcd010078bab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.273128 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.271251    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f1ee110b-ab37-4c60-b72c-74f07b55668d-agent-certs\") pod \"konnectivity-agent-wk6ks\" (UID: \"f1ee110b-ab37-4c60-b72c-74f07b55668d\") " pod="kube-system/konnectivity-agent-wk6ks"
Aug 05 17:12:31.273128 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.271564    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77260dfe-e37a-487f-8855-324bbed53c7a-etc-tuned\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.273128 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.271756    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/03b7f246-02d1-4207-94ff-1264fb006a98-ovn-node-metrics-cert\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.282845 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.282820    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbzbl\" (UniqueName: \"kubernetes.io/projected/fc979956-0bb5-4600-81bb-19a5521be56d-kube-api-access-tbzbl\") pod \"node-resolver-kgdd2\" (UID: \"fc979956-0bb5-4600-81bb-19a5521be56d\") " pod="openshift-dns/node-resolver-kgdd2"
Aug 05 17:12:31.283023 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.283007    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hkrg\" (UniqueName: \"kubernetes.io/projected/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-kube-api-access-9hkrg\") pod \"network-metrics-daemon-xwnbn\" (UID: \"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660\") " pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:31.288596 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.288580    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlljz\" (UniqueName: \"kubernetes.io/projected/03b7f246-02d1-4207-94ff-1264fb006a98-kube-api-access-hlljz\") pod \"ovnkube-node-xrhbh\" (UID: \"03b7f246-02d1-4207-94ff-1264fb006a98\") " pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.291273 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.291249    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grp48\" (UniqueName: \"kubernetes.io/projected/6e6fb427-9f41-42ec-9c75-c9c49caebf28-kube-api-access-grp48\") pod \"aws-ebs-csi-driver-node-rsdhd\" (UID: \"6e6fb427-9f41-42ec-9c75-c9c49caebf28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.291998 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.291977    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:31.292093 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.292002    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:31.292093 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.292015    2574 projected.go:196] Error preparing data for projected volume kube-api-access-8nn8v for pod openshift-network-diagnostics/network-check-target-44mzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:31.292093 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.292066    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtwhr\" (UniqueName: \"kubernetes.io/projected/5286a5af-946f-4116-8ec5-33c3d9dbafb7-kube-api-access-gtwhr\") pod \"iptables-alerter-ldtgp\" (UID: \"5286a5af-946f-4116-8ec5-33c3d9dbafb7\") " pod="openshift-network-operator/iptables-alerter-ldtgp"
Aug 05 17:12:31.292226 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.292102    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v podName:1fc45fe8-5a2b-4e42-bf7e-938c50805465 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:31.792068412 +0000 UTC m=+2.083923243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8nn8v" (UniqueName: "kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v") pod "network-check-target-44mzx" (UID: "1fc45fe8-5a2b-4e42-bf7e-938c50805465") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:31.292664 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.292643    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swj8q\" (UniqueName: \"kubernetes.io/projected/4a578ff9-e0d4-4386-a305-bcd010078bab-kube-api-access-swj8q\") pod \"multus-additional-cni-plugins-n6mhh\" (UID: \"4a578ff9-e0d4-4386-a305-bcd010078bab\") " pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.293026 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.293004    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c58t9\" (UniqueName: \"kubernetes.io/projected/2347bfcd-6da3-456d-a788-a18eaf3dbb61-kube-api-access-c58t9\") pod \"multus-2slm7\" (UID: \"2347bfcd-6da3-456d-a788-a18eaf3dbb61\") " pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.294005 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.293990    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnrtp\" (UniqueName: \"kubernetes.io/projected/ef418180-36b0-450a-8ac5-8518f605f717-kube-api-access-jnrtp\") pod \"node-ca-hsn68\" (UID: \"ef418180-36b0-450a-8ac5-8518f605f717\") " pod="openshift-image-registry/node-ca-hsn68"
Aug 05 17:12:31.294292 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.294275    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtlgs\" (UniqueName: \"kubernetes.io/projected/77260dfe-e37a-487f-8855-324bbed53c7a-kube-api-access-mtlgs\") pod \"tuned-b78nx\" (UID: \"77260dfe-e37a-487f-8855-324bbed53c7a\") " pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.381932 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.381843    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b78nx"
Aug 05 17:12:31.389856 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:31.389831    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77260dfe_e37a_487f_8855_324bbed53c7a.slice/crio-a6e931cd6fd50252242f62b20ab4969ac28abb325c481655756b0e75dd7c4985 WatchSource:0}: Error finding container a6e931cd6fd50252242f62b20ab4969ac28abb325c481655756b0e75dd7c4985: Status 404 returned error can't find the container with id a6e931cd6fd50252242f62b20ab4969ac28abb325c481655756b0e75dd7c4985
Aug 05 17:12:31.403970 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.403949    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kgdd2"
Aug 05 17:12:31.411495 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:31.411466    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc979956_0bb5_4600_81bb_19a5521be56d.slice/crio-f31665f10b748ad4c3e61cd35093f7ce370dd1f3a41d7bf002cdcdf0eb0a7855 WatchSource:0}: Error finding container f31665f10b748ad4c3e61cd35093f7ce370dd1f3a41d7bf002cdcdf0eb0a7855: Status 404 returned error can't find the container with id f31665f10b748ad4c3e61cd35093f7ce370dd1f3a41d7bf002cdcdf0eb0a7855
Aug 05 17:12:31.417251 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.417233    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hsn68"
Aug 05 17:12:31.421886 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.421870    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ldtgp"
Aug 05 17:12:31.423789 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:31.423763    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef418180_36b0_450a_8ac5_8518f605f717.slice/crio-72f485655d333ed624fa8e3838fbde848b246abd56a5f05baf0695a8b983f68a WatchSource:0}: Error finding container 72f485655d333ed624fa8e3838fbde848b246abd56a5f05baf0695a8b983f68a: Status 404 returned error can't find the container with id 72f485655d333ed624fa8e3838fbde848b246abd56a5f05baf0695a8b983f68a
Aug 05 17:12:31.429705 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:31.429676    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5286a5af_946f_4116_8ec5_33c3d9dbafb7.slice/crio-3185a2f35aa8998569945a5c7a1d618303b2b185a996806794a7e30489e9dd7d WatchSource:0}: Error finding container 3185a2f35aa8998569945a5c7a1d618303b2b185a996806794a7e30489e9dd7d: Status 404 returned error can't find the container with id 3185a2f35aa8998569945a5c7a1d618303b2b185a996806794a7e30489e9dd7d
Aug 05 17:12:31.441553 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.441532    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd"
Aug 05 17:12:31.449018 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:31.448993    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6fb427_9f41_42ec_9c75_c9c49caebf28.slice/crio-1d96040c7477820d1d523560d9f74d3a34b43ceb63a3475d9506c6feb7fd1b15 WatchSource:0}: Error finding container 1d96040c7477820d1d523560d9f74d3a34b43ceb63a3475d9506c6feb7fd1b15: Status 404 returned error can't find the container with id 1d96040c7477820d1d523560d9f74d3a34b43ceb63a3475d9506c6feb7fd1b15
Aug 05 17:12:31.459465 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.459446    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2slm7"
Aug 05 17:12:31.473067 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.473044    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n6mhh"
Aug 05 17:12:31.480282 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:31.480258    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a578ff9_e0d4_4386_a305_bcd010078bab.slice/crio-5958cdacfd33ec44195a0dbf1b1907fa6591eee7856140e6003ff706386bb704 WatchSource:0}: Error finding container 5958cdacfd33ec44195a0dbf1b1907fa6591eee7856140e6003ff706386bb704: Status 404 returned error can't find the container with id 5958cdacfd33ec44195a0dbf1b1907fa6591eee7856140e6003ff706386bb704
Aug 05 17:12:31.481465 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.481389    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:31.485835 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.485817    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wk6ks"
Aug 05 17:12:31.488780 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:31.488759    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b7f246_02d1_4207_94ff_1264fb006a98.slice/crio-fcbae51be1fff684bc876270b6168f11784d3c1ff5727b199f53b05deab71ba4 WatchSource:0}: Error finding container fcbae51be1fff684bc876270b6168f11784d3c1ff5727b199f53b05deab71ba4: Status 404 returned error can't find the container with id fcbae51be1fff684bc876270b6168f11784d3c1ff5727b199f53b05deab71ba4
Aug 05 17:12:31.493694 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:12:31.493673    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ee110b_ab37_4c60_b72c_74f07b55668d.slice/crio-1a2a6478e3f821864d22258f9d05e93128e5a593ccef0737d286c8100653fd58 WatchSource:0}: Error finding container 1a2a6478e3f821864d22258f9d05e93128e5a593ccef0737d286c8100653fd58: Status 404 returned error can't find the container with id 1a2a6478e3f821864d22258f9d05e93128e5a593ccef0737d286c8100653fd58
Aug 05 17:12:31.534196 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.534169    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:31.543615 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.543594    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:31.773164 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.773129    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:31.773329 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.773207    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs\") pod \"network-metrics-daemon-xwnbn\" (UID: \"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660\") " pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:31.773329 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.773284    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:31.773400 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.773343    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret podName:057a0fc7-bbc2-47e0-9b9f-d4e292b354f1 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:32.773329746 +0000 UTC m=+3.065184558 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret") pod "global-pull-secret-syncer-w89xx" (UID: "057a0fc7-bbc2-47e0-9b9f-d4e292b354f1") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:31.773400 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.773284    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:31.773470 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.773412    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs podName:a7531d8b-5f4c-43e0-bc2f-d9e2a415f660 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:32.77339975 +0000 UTC m=+3.065254561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs") pod "network-metrics-daemon-xwnbn" (UID: "a7531d8b-5f4c-43e0-bc2f-d9e2a415f660") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:31.873879 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:31.873842    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nn8v\" (UniqueName: \"kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v\") pod \"network-check-target-44mzx\" (UID: \"1fc45fe8-5a2b-4e42-bf7e-938c50805465\") " pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:31.874087 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.874033    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:31.874087 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.874059    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:31.874087 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.874087    2574 projected.go:196] Error preparing data for projected volume kube-api-access-8nn8v for pod openshift-network-diagnostics/network-check-target-44mzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:31.874259 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:31.874154    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v podName:1fc45fe8-5a2b-4e42-bf7e-938c50805465 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:32.874134811 +0000 UTC m=+3.165989623 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8nn8v" (UniqueName: "kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v") pod "network-check-target-44mzx" (UID: "1fc45fe8-5a2b-4e42-bf7e-938c50805465") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:32.135136 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.134954    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 17:07:31 +0000 UTC" deadline="2028-01-08 12:50:26.320187511 +0000 UTC"
Aug 05 17:12:32.135136 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.135014    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12499h37m54.185177307s"
Aug 05 17:12:32.191694 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.191663    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:32.191863 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:32.191819    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:32.212333 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.212220    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2slm7" event={"ID":"2347bfcd-6da3-456d-a788-a18eaf3dbb61","Type":"ContainerStarted","Data":"438599f07874f6a5a788575e39f5f2d5a69cf05ac93fcc56263d7dd127041789"}
Aug 05 17:12:32.239964 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.239924    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd" event={"ID":"6e6fb427-9f41-42ec-9c75-c9c49caebf28","Type":"ContainerStarted","Data":"1d96040c7477820d1d523560d9f74d3a34b43ceb63a3475d9506c6feb7fd1b15"}
Aug 05 17:12:32.247658 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.247620    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kgdd2" event={"ID":"fc979956-0bb5-4600-81bb-19a5521be56d","Type":"ContainerStarted","Data":"f31665f10b748ad4c3e61cd35093f7ce370dd1f3a41d7bf002cdcdf0eb0a7855"}
Aug 05 17:12:32.252934 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.252898    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b78nx" event={"ID":"77260dfe-e37a-487f-8855-324bbed53c7a","Type":"ContainerStarted","Data":"a6e931cd6fd50252242f62b20ab4969ac28abb325c481655756b0e75dd7c4985"}
Aug 05 17:12:32.256650 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.256618    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ldtgp" event={"ID":"5286a5af-946f-4116-8ec5-33c3d9dbafb7","Type":"ContainerStarted","Data":"3185a2f35aa8998569945a5c7a1d618303b2b185a996806794a7e30489e9dd7d"}
Aug 05 17:12:32.269802 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.269754    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hsn68" event={"ID":"ef418180-36b0-450a-8ac5-8518f605f717","Type":"ContainerStarted","Data":"72f485655d333ed624fa8e3838fbde848b246abd56a5f05baf0695a8b983f68a"}
Aug 05 17:12:32.274976 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.274944    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-192.ec2.internal" event={"ID":"322b4eab99b8f4f207438c9174b617a8","Type":"ContainerStarted","Data":"3a0df816195495cd310f4c7e40b8f98f31ef1d9a92323d64ee4af9d18b669157"}
Aug 05 17:12:32.280352 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.280319    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal" event={"ID":"36f6ab0e8ee2c59be4633f740c4f4bb8","Type":"ContainerStarted","Data":"0e8f7b24665ff901ee85b98391c79bc1701b41d65f0ba7261c717f8f1aff1cd5"}
Aug 05 17:12:32.289434 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.289333    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wk6ks" event={"ID":"f1ee110b-ab37-4c60-b72c-74f07b55668d","Type":"ContainerStarted","Data":"1a2a6478e3f821864d22258f9d05e93128e5a593ccef0737d286c8100653fd58"}
Aug 05 17:12:32.301851 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.301815    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh" event={"ID":"03b7f246-02d1-4207-94ff-1264fb006a98","Type":"ContainerStarted","Data":"fcbae51be1fff684bc876270b6168f11784d3c1ff5727b199f53b05deab71ba4"}
Aug 05 17:12:32.325228 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.325188    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6mhh" event={"ID":"4a578ff9-e0d4-4386-a305-bcd010078bab","Type":"ContainerStarted","Data":"5958cdacfd33ec44195a0dbf1b1907fa6591eee7856140e6003ff706386bb704"}
Aug 05 17:12:32.510114 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.509864    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:32.780979 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.780833    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs\") pod \"network-metrics-daemon-xwnbn\" (UID: \"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660\") " pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:32.780979 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.780888    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:32.781231 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:32.781038    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:32.781231 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:32.781118    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret podName:057a0fc7-bbc2-47e0-9b9f-d4e292b354f1 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:34.781099307 +0000 UTC m=+5.072954118 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret") pod "global-pull-secret-syncer-w89xx" (UID: "057a0fc7-bbc2-47e0-9b9f-d4e292b354f1") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:32.781534 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:32.781517    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:32.781625 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:32.781571    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs podName:a7531d8b-5f4c-43e0-bc2f-d9e2a415f660 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:34.781554437 +0000 UTC m=+5.073409260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs") pod "network-metrics-daemon-xwnbn" (UID: "a7531d8b-5f4c-43e0-bc2f-d9e2a415f660") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:32.882867 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:32.882106    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nn8v\" (UniqueName: \"kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v\") pod \"network-check-target-44mzx\" (UID: \"1fc45fe8-5a2b-4e42-bf7e-938c50805465\") " pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:32.882867 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:32.882349    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:32.882867 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:32.882372    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:32.882867 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:32.882386    2574 projected.go:196] Error preparing data for projected volume kube-api-access-8nn8v for pod openshift-network-diagnostics/network-check-target-44mzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:32.882867 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:32.882446    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v podName:1fc45fe8-5a2b-4e42-bf7e-938c50805465 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:34.882428633 +0000 UTC m=+5.174283448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8nn8v" (UniqueName: "kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v") pod "network-check-target-44mzx" (UID: "1fc45fe8-5a2b-4e42-bf7e-938c50805465") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:33.136117 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:33.135979    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 17:07:31 +0000 UTC" deadline="2028-01-02 09:51:54.675207663 +0000 UTC"
Aug 05 17:12:33.136117 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:33.136019    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12352h39m21.539192749s"
Aug 05 17:12:33.189297 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:33.189262    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:33.189478 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:33.189404    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:12:33.189806 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:33.189784    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:33.189910 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:33.189881    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:12:33.767656 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:33.767425    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:34.191086 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:34.190965    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:34.191449 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:34.191114    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:34.797478 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:34.796696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs\") pod \"network-metrics-daemon-xwnbn\" (UID: \"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660\") " pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:34.797478 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:34.796757    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:34.797478 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:34.796907    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:34.797478 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:34.796968    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret podName:057a0fc7-bbc2-47e0-9b9f-d4e292b354f1 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:38.796949905 +0000 UTC m=+9.088804728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret") pod "global-pull-secret-syncer-w89xx" (UID: "057a0fc7-bbc2-47e0-9b9f-d4e292b354f1") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:34.797478 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:34.797395    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:34.797478 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:34.797443    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs podName:a7531d8b-5f4c-43e0-bc2f-d9e2a415f660 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:38.79742824 +0000 UTC m=+9.089283054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs") pod "network-metrics-daemon-xwnbn" (UID: "a7531d8b-5f4c-43e0-bc2f-d9e2a415f660") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:34.897585 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:34.897548    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nn8v\" (UniqueName: \"kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v\") pod \"network-check-target-44mzx\" (UID: \"1fc45fe8-5a2b-4e42-bf7e-938c50805465\") " pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:34.897774 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:34.897717    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:34.897774 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:34.897743    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:34.897774 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:34.897759    2574 projected.go:196] Error preparing data for projected volume kube-api-access-8nn8v for pod openshift-network-diagnostics/network-check-target-44mzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:34.897938 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:34.897822    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v podName:1fc45fe8-5a2b-4e42-bf7e-938c50805465 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:38.897801231 +0000 UTC m=+9.189656054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8nn8v" (UniqueName: "kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v") pod "network-check-target-44mzx" (UID: "1fc45fe8-5a2b-4e42-bf7e-938c50805465") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:35.188801 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:35.188723    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:35.188801 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:35.188770    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:35.189018 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:35.188856    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:12:35.189101 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:35.189056    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:12:36.188737 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:36.188705    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:36.189248 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:36.188844    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:37.188948 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:37.188864    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:37.189575 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:37.188874    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:37.189575 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:37.188996    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:12:37.189575 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:37.189102    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:12:38.191878 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:38.191840    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:38.192369 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:38.191982    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:38.834028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:38.833326    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs\") pod \"network-metrics-daemon-xwnbn\" (UID: \"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660\") " pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:38.834028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:38.833382    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:38.834028 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:38.833519    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:38.834028 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:38.833578    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret podName:057a0fc7-bbc2-47e0-9b9f-d4e292b354f1 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:46.833561158 +0000 UTC m=+17.125415979 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret") pod "global-pull-secret-syncer-w89xx" (UID: "057a0fc7-bbc2-47e0-9b9f-d4e292b354f1") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:38.834028 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:38.833953    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:38.834028 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:38.833994    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs podName:a7531d8b-5f4c-43e0-bc2f-d9e2a415f660 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:46.833980328 +0000 UTC m=+17.125835151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs") pod "network-metrics-daemon-xwnbn" (UID: "a7531d8b-5f4c-43e0-bc2f-d9e2a415f660") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:38.934272 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:38.934240    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nn8v\" (UniqueName: \"kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v\") pod \"network-check-target-44mzx\" (UID: \"1fc45fe8-5a2b-4e42-bf7e-938c50805465\") " pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:38.934458 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:38.934400    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:38.934458 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:38.934423    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:38.934458 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:38.934436    2574 projected.go:196] Error preparing data for projected volume kube-api-access-8nn8v for pod openshift-network-diagnostics/network-check-target-44mzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:38.934622 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:38.934542    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v podName:1fc45fe8-5a2b-4e42-bf7e-938c50805465 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:46.93452289 +0000 UTC m=+17.226377743 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8nn8v" (UniqueName: "kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v") pod "network-check-target-44mzx" (UID: "1fc45fe8-5a2b-4e42-bf7e-938c50805465") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:39.189720 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:39.189276    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:39.189720 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:39.189313    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:39.189720 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:39.189422    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:12:39.189720 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:39.189556    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:12:40.190578 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:40.190544    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:40.191021 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:40.190661    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:41.188989 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:41.188957    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:41.189250 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:41.189106    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:12:41.189250 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:41.189132    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:41.189250 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:41.189226    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:12:42.188959 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:42.188930    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:42.189367 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:42.189045    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:43.188795 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:43.188702    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:43.188795 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:43.188750    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:43.189001 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:43.188832    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:12:43.189001 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:43.188943    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:12:44.189423 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:44.189387    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:44.189861 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:44.189532    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:45.188677 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:45.188639    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:45.188863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:45.188639    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:45.188863 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:45.188761    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:12:45.188979 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:45.188894    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:12:46.189203 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:46.189170    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:46.189618 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:46.189311    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:46.896534 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:46.896488    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs\") pod \"network-metrics-daemon-xwnbn\" (UID: \"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660\") " pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:46.896781 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:46.896557    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:46.896781 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:46.896635    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:46.896781 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:46.896679    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:46.896781 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:46.896721    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs podName:a7531d8b-5f4c-43e0-bc2f-d9e2a415f660 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:02.896700373 +0000 UTC m=+33.188555193 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs") pod "network-metrics-daemon-xwnbn" (UID: "a7531d8b-5f4c-43e0-bc2f-d9e2a415f660") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:46.896781 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:46.896739    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret podName:057a0fc7-bbc2-47e0-9b9f-d4e292b354f1 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:02.896731747 +0000 UTC m=+33.188586561 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret") pod "global-pull-secret-syncer-w89xx" (UID: "057a0fc7-bbc2-47e0-9b9f-d4e292b354f1") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:46.997222 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:46.997187    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nn8v\" (UniqueName: \"kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v\") pod \"network-check-target-44mzx\" (UID: \"1fc45fe8-5a2b-4e42-bf7e-938c50805465\") " pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:46.997414 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:46.997392    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:46.997460 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:46.997421    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:46.997460 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:46.997436    2574 projected.go:196] Error preparing data for projected volume kube-api-access-8nn8v for pod openshift-network-diagnostics/network-check-target-44mzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:46.997522 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:46.997501    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v podName:1fc45fe8-5a2b-4e42-bf7e-938c50805465 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:02.997481438 +0000 UTC m=+33.289336266 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8nn8v" (UniqueName: "kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v") pod "network-check-target-44mzx" (UID: "1fc45fe8-5a2b-4e42-bf7e-938c50805465") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:47.188858 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:47.188776    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:47.189007 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:47.188775    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:47.189007 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:47.188915    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:12:47.189138 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:47.189012    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:12:48.189294 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:48.189264    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:48.189685 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:48.189367    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:49.189597 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:49.189557    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:49.189967 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:49.189557    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:49.189967 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:49.189693    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:12:49.189967 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:49.189730    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:12:50.189663 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:50.189627    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:50.190130 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:50.189725    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:51.189617 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:51.189583    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:51.189783 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:51.189593    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:51.189783 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:51.189738    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:12:51.190209 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:51.189825    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:12:52.189899 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.189670    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:52.190530 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:52.190023    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:52.377044 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.377009    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wk6ks" event={"ID":"f1ee110b-ab37-4c60-b72c-74f07b55668d","Type":"ContainerStarted","Data":"b35d2e297506486cee523d010a474f2a5546d79e94167886260db86ffbcb8636"}
Aug 05 17:12:52.379589 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.379556    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh" event={"ID":"03b7f246-02d1-4207-94ff-1264fb006a98","Type":"ContainerStarted","Data":"07cb8ecc27eba01c34b56ec9d904c818b2b5c824349b99ace06e022729fe9fe1"}
Aug 05 17:12:52.379727 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.379642    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh" event={"ID":"03b7f246-02d1-4207-94ff-1264fb006a98","Type":"ContainerStarted","Data":"abcf7156db114ce787ec8bd3367a8c7b8510dd6b3231e2d0dcb44c52c77da766"}
Aug 05 17:12:52.379727 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.379695    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh" event={"ID":"03b7f246-02d1-4207-94ff-1264fb006a98","Type":"ContainerStarted","Data":"c54d5fcdb3c389cb501d20a8b08b08c0dd2f8f7ac9c6a6c957723d51dafe1abe"}
Aug 05 17:12:52.379951 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.379745    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh" event={"ID":"03b7f246-02d1-4207-94ff-1264fb006a98","Type":"ContainerStarted","Data":"eb3fc6637f092ec42ed4d0466756bc611f16aefafb6d7f5cc52e955db5a053de"}
Aug 05 17:12:52.381033 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.381010    2574 generic.go:356] "Generic (PLEG): container finished" podID="4a578ff9-e0d4-4386-a305-bcd010078bab" containerID="dd095e7d540553bfb5a4f8ba4d2de6766ea9b8f3602be284e82021a13822ae25" exitCode=0
Aug 05 17:12:52.381189 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.381112    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6mhh" event={"ID":"4a578ff9-e0d4-4386-a305-bcd010078bab","Type":"ContainerDied","Data":"dd095e7d540553bfb5a4f8ba4d2de6766ea9b8f3602be284e82021a13822ae25"}
Aug 05 17:12:52.382726 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.382699    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2slm7" event={"ID":"2347bfcd-6da3-456d-a788-a18eaf3dbb61","Type":"ContainerStarted","Data":"a5718e090778e252e2ec7135d1ca53b43bf16f09dc1f84106e4cb9d1ffe3b46e"}
Aug 05 17:12:52.384204 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.384154    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd" event={"ID":"6e6fb427-9f41-42ec-9c75-c9c49caebf28","Type":"ContainerStarted","Data":"a97abd932589b00b96e3e6ab041e4f60917f8bc4f6ec08f3b986a9aabac55cec"}
Aug 05 17:12:52.385484 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.385461    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kgdd2" event={"ID":"fc979956-0bb5-4600-81bb-19a5521be56d","Type":"ContainerStarted","Data":"3771b16fef8f907340bdd031e74b9f8fba083e8185e2d0029833c214efd3bd26"}
Aug 05 17:12:52.386944 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.386922    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b78nx" event={"ID":"77260dfe-e37a-487f-8855-324bbed53c7a","Type":"ContainerStarted","Data":"f5a7cf1f690bfe11773723c17fb54364f19598bd655557fcc5579e9f12e9675c"}
Aug 05 17:12:52.388434 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.388414    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hsn68" event={"ID":"ef418180-36b0-450a-8ac5-8518f605f717","Type":"ContainerStarted","Data":"cbc08fa296fca5d6d9b5b905d21ce9e97c18038367632362672cb9bf4641c2fd"}
Aug 05 17:12:52.389619 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.389601    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-192.ec2.internal" event={"ID":"322b4eab99b8f4f207438c9174b617a8","Type":"ContainerStarted","Data":"91c3b041429534d89ab0bb3db196a8055e0121840a6c76533c5f225888a4f11d"}
Aug 05 17:12:52.391401 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.391381    2574 generic.go:356] "Generic (PLEG): container finished" podID="36f6ab0e8ee2c59be4633f740c4f4bb8" containerID="c99c152a78c3c2bd9ddc3972c6525e022d37c3b224be06a48cd7829ed49d93b1" exitCode=0
Aug 05 17:12:52.391476 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.391409    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal" event={"ID":"36f6ab0e8ee2c59be4633f740c4f4bb8","Type":"ContainerDied","Data":"c99c152a78c3c2bd9ddc3972c6525e022d37c3b224be06a48cd7829ed49d93b1"}
Aug 05 17:12:52.393232 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.393187    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wk6ks" podStartSLOduration=2.209382985 podStartE2EDuration="22.39317628s" totalImagesPullingTime="20.183793295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:52.393050336 +0000 UTC m=+22.684905167" watchObservedRunningTime="2026-08-05 17:12:52.39317628 +0000 UTC m=+22.685031111"
Aug 05 17:12:52.425068 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.425026    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hsn68" podStartSLOduration=2.171991031 podStartE2EDuration="22.425010698s" totalImagesPullingTime="20.253019667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:52.424746043 +0000 UTC m=+22.716600886" watchObservedRunningTime="2026-08-05 17:12:52.425010698 +0000 UTC m=+22.716865561"
Aug 05 17:12:52.440103 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.440047    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-192.ec2.internal" podStartSLOduration=21.440031716 podStartE2EDuration="21.440031716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:52.439381183 +0000 UTC m=+22.731236015" watchObservedRunningTime="2026-08-05 17:12:52.440031716 +0000 UTC m=+22.731886547"
Aug 05 17:12:52.458547 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.458505    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b78nx" podStartSLOduration=2.133908632 podStartE2EDuration="22.458488847s" totalImagesPullingTime="20.324580215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:52.45829118 +0000 UTC m=+22.750146011" watchObservedRunningTime="2026-08-05 17:12:52.458488847 +0000 UTC m=+22.750343678"
Aug 05 17:12:52.519107 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.519037    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kgdd2" podStartSLOduration=2.327664527 podStartE2EDuration="22.519021244s" totalImagesPullingTime="20.191356717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:52.488761948 +0000 UTC m=+22.780616779" watchObservedRunningTime="2026-08-05 17:12:52.519021244 +0000 UTC m=+22.810876075"
Aug 05 17:12:52.519329 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.519303    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2slm7" podStartSLOduration=2.120125038 podStartE2EDuration="22.519297447s" totalImagesPullingTime="20.399172409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:52.518830316 +0000 UTC m=+22.810685146" watchObservedRunningTime="2026-08-05 17:12:52.519297447 +0000 UTC m=+22.811152277"
Aug 05 17:12:52.779013 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.778982    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wk6ks"
Aug 05 17:12:52.779547 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:52.779527    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wk6ks"
Aug 05 17:12:53.188817 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:53.188783    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:53.189008 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:53.188980    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:53.189097 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:53.189023    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:12:53.189097 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:53.189089    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:12:53.396776 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:53.396692    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh" event={"ID":"03b7f246-02d1-4207-94ff-1264fb006a98","Type":"ContainerStarted","Data":"180636936e0f511c5740dedc4e25773973a0629a3667cb494940a10a646a23dc"}
Aug 05 17:12:53.396776 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:53.396773    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh" event={"ID":"03b7f246-02d1-4207-94ff-1264fb006a98","Type":"ContainerStarted","Data":"af1d61f4a7f5f22d64104f6bc18ca889850e3d2d23bd12dd052aeed8e64dfb19"}
Aug 05 17:12:53.398757 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:53.398726    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ldtgp" event={"ID":"5286a5af-946f-4116-8ec5-33c3d9dbafb7","Type":"ContainerStarted","Data":"4c342b2f1f9ad3123673e16fb7b66e3efc25aa92d8cc03b880d3b9d8987781ee"}
Aug 05 17:12:53.399109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:53.399087    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wk6ks"
Aug 05 17:12:53.399501 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:53.399480    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wk6ks"
Aug 05 17:12:53.759419 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:53.759396    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 05 17:12:54.126030 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:54.125924    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-05T17:12:53.759413511Z","UUID":"a6a9694f-89c8-4651-9dea-c6d80666e98a","Handler":null,"Name":"","Endpoint":""}
Aug 05 17:12:54.128391 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:54.128333    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 05 17:12:54.128391 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:54.128380    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 05 17:12:54.189666 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:54.189636    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:54.189845 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:54.189772    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:54.403407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:54.403314    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd" event={"ID":"6e6fb427-9f41-42ec-9c75-c9c49caebf28","Type":"ContainerStarted","Data":"59c03587d130bb5f7f7fbf8b8a3d7c4fb9cbd21098d45baafd562c7d54106ab2"}
Aug 05 17:12:54.405422 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:54.405395    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal" event={"ID":"36f6ab0e8ee2c59be4633f740c4f4bb8","Type":"ContainerStarted","Data":"2a77fc338c9553a77eba74d51deb1098ee4ef34f4650d89ac6334a7282ae6290"}
Aug 05 17:12:54.437945 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:54.437875    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ldtgp" podStartSLOduration=4.152311595 podStartE2EDuration="24.437853272s" totalImagesPullingTime="20.285541677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:53.471060998 +0000 UTC m=+23.762915832" watchObservedRunningTime="2026-08-05 17:12:54.437853272 +0000 UTC m=+24.729708104"
Aug 05 17:12:54.438592 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:54.438550    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-192.ec2.internal" podStartSLOduration=23.438534429 podStartE2EDuration="23.438534429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:54.437787619 +0000 UTC m=+24.729642464" watchObservedRunningTime="2026-08-05 17:12:54.438534429 +0000 UTC m=+24.730389262"
Aug 05 17:12:55.189048 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:55.189008    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:55.189239 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:55.189156    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:12:55.189239 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:55.189186    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:55.189347 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:55.189282    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:12:55.409886 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:55.409839    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd" event={"ID":"6e6fb427-9f41-42ec-9c75-c9c49caebf28","Type":"ContainerStarted","Data":"3bdaf1ecc4c64ac29e5577767b16a28d399850c07f486ba417647fd21a740a9b"}
Aug 05 17:12:55.413658 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:55.413617    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh" event={"ID":"03b7f246-02d1-4207-94ff-1264fb006a98","Type":"ContainerStarted","Data":"b53d2b9f24eb77ecb3af8b7c7ac2483e324fe00d44bd70fa342a57a02e50e1e4"}
Aug 05 17:12:55.436701 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:55.436636    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsdhd" podStartSLOduration=2.986976599 podStartE2EDuration="25.436612096s" totalImagesPullingTime="22.449635497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:55.431440006 +0000 UTC m=+25.723294837" watchObservedRunningTime="2026-08-05 17:12:55.436612096 +0000 UTC m=+25.728466929"
Aug 05 17:12:56.192827 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:56.192796    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:56.192989 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:56.192907    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:57.188836 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:57.188625    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:57.189482 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:57.188624    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:57.189482 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:57.188976    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:12:57.189482 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:57.189102    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:12:57.422120 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:57.422060    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh" event={"ID":"03b7f246-02d1-4207-94ff-1264fb006a98","Type":"ContainerStarted","Data":"ad58bfbc8df075232c98fc4fe482e750c59e2ae21e6ba61719d9ec053ec3ade9"}
Aug 05 17:12:57.422347 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:57.422329    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:57.424093 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:57.424048    2574 generic.go:356] "Generic (PLEG): container finished" podID="4a578ff9-e0d4-4386-a305-bcd010078bab" containerID="d43cffeb0fdaf25c8204e7afe3769adc3fa6d6008d898148a56d81a027986a49" exitCode=0
Aug 05 17:12:57.424211 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:57.424114    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6mhh" event={"ID":"4a578ff9-e0d4-4386-a305-bcd010078bab","Type":"ContainerDied","Data":"d43cffeb0fdaf25c8204e7afe3769adc3fa6d6008d898148a56d81a027986a49"}
Aug 05 17:12:57.438471 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:57.438446    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:57.456385 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:57.456290    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh" podStartSLOduration=7.209623972 podStartE2EDuration="27.456275294s" totalImagesPullingTime="20.246651322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:57.455512252 +0000 UTC m=+27.747367084" watchObservedRunningTime="2026-08-05 17:12:57.456275294 +0000 UTC m=+27.748130125"
Aug 05 17:12:58.189378 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:58.189128    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:58.189881 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:58.189488    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:58.427044 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:58.426967    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:58.427206 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:58.427059    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:58.442041 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:58.442012    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:12:59.001847 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:59.001813    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-44mzx"]
Aug 05 17:12:59.002041 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:59.001948    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:12:59.002136 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:59.002067    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:12:59.004854 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:59.004829    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w89xx"]
Aug 05 17:12:59.004985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:59.004922    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:12:59.005045 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:59.005017    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:12:59.007243 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:59.007219    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xwnbn"]
Aug 05 17:12:59.007361 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:59.007336    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:12:59.007438 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:12:59.007420    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:12:59.430871 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:59.430784    2574 generic.go:356] "Generic (PLEG): container finished" podID="4a578ff9-e0d4-4386-a305-bcd010078bab" containerID="9f162f65fc93ee8a4e2dd5c61f0f483167d509dbd40fbaca76fc1713e70c3951" exitCode=0
Aug 05 17:12:59.431289 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:12:59.430871    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6mhh" event={"ID":"4a578ff9-e0d4-4386-a305-bcd010078bab","Type":"ContainerDied","Data":"9f162f65fc93ee8a4e2dd5c61f0f483167d509dbd40fbaca76fc1713e70c3951"}
Aug 05 17:13:00.189999 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:00.189967    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:13:00.190229 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:00.190093    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:13:01.189332 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:01.189248    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:13:01.189332 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:01.189281    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:13:01.189777 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:01.189383    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:13:01.190330 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:01.189953    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:13:01.437630 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:01.437551    2574 generic.go:356] "Generic (PLEG): container finished" podID="4a578ff9-e0d4-4386-a305-bcd010078bab" containerID="e73af2d9a2a4ec8ac75f4552822eed07637103657d6542c36becce20b4217cfd" exitCode=0
Aug 05 17:13:01.437630 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:01.437606    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6mhh" event={"ID":"4a578ff9-e0d4-4386-a305-bcd010078bab","Type":"ContainerDied","Data":"e73af2d9a2a4ec8ac75f4552822eed07637103657d6542c36becce20b4217cfd"}
Aug 05 17:13:02.189173 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:02.189134    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:13:02.189398 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:02.189303    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:13:02.921311 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:02.921274    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:13:02.921480 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:02.921364    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs\") pod \"network-metrics-daemon-xwnbn\" (UID: \"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660\") " pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:13:02.921480 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:02.921405    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:13:02.921480 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:02.921461    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:13:02.921631 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:02.921485    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret podName:057a0fc7-bbc2-47e0-9b9f-d4e292b354f1 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:34.921462431 +0000 UTC m=+65.213317251 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret") pod "global-pull-secret-syncer-w89xx" (UID: "057a0fc7-bbc2-47e0-9b9f-d4e292b354f1") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:13:02.921631 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:02.921509    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs podName:a7531d8b-5f4c-43e0-bc2f-d9e2a415f660 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:34.921493132 +0000 UTC m=+65.213347956 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs") pod "network-metrics-daemon-xwnbn" (UID: "a7531d8b-5f4c-43e0-bc2f-d9e2a415f660") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:13:03.022197 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:03.022154    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nn8v\" (UniqueName: \"kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v\") pod \"network-check-target-44mzx\" (UID: \"1fc45fe8-5a2b-4e42-bf7e-938c50805465\") " pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:13:03.022408 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:03.022391    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:13:03.022459 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:03.022415    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:13:03.022499 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:03.022463    2574 projected.go:196] Error preparing data for projected volume kube-api-access-8nn8v for pod openshift-network-diagnostics/network-check-target-44mzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:13:03.022534 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:03.022520    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v podName:1fc45fe8-5a2b-4e42-bf7e-938c50805465 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:35.02250376 +0000 UTC m=+65.314358570 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8nn8v" (UniqueName: "kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v") pod "network-check-target-44mzx" (UID: "1fc45fe8-5a2b-4e42-bf7e-938c50805465") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:13:03.189559 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:03.189473    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:13:03.190010 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:03.189482    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:13:03.190010 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:03.189623    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w89xx" podUID="057a0fc7-bbc2-47e0-9b9f-d4e292b354f1"
Aug 05 17:13:03.190010 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:03.189668    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44mzx" podUID="1fc45fe8-5a2b-4e42-bf7e-938c50805465"
Aug 05 17:13:04.189279 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.189247    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:13:04.189474 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:04.189368    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:13:04.543857 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.543664    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-192.ec2.internal" event="NodeReady"
Aug 05 17:13:04.544313 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.543986    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 05 17:13:04.589837 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.589803    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv"]
Aug 05 17:13:04.592387 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.592361    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv"
Aug 05 17:13:04.595270 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.595249    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-mj92t\""
Aug 05 17:13:04.595402 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.595369    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Aug 05 17:13:04.596014 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.595984    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Aug 05 17:13:04.596120 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.596024    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Aug 05 17:13:04.596120 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.596094    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Aug 05 17:13:04.601447 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.600907    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"]
Aug 05 17:13:04.602878 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.602861    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"]
Aug 05 17:13:04.603016 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.602999    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.604468 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.604450    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"]
Aug 05 17:13:04.604594 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.604572    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:13:04.606162 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.606143    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Aug 05 17:13:04.606162 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.606169    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Aug 05 17:13:04.606331 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.606195    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Aug 05 17:13:04.606386 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.606349    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv"]
Aug 05 17:13:04.606446 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.606430    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.606493 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.606452    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Aug 05 17:13:04.606970 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.606953    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Aug 05 17:13:04.609305 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.609285    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Aug 05 17:13:04.609566 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.609545    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Aug 05 17:13:04.609651 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.609582    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xrzw4\""
Aug 05 17:13:04.609651 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.609634    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Aug 05 17:13:04.615534 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.615337    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Aug 05 17:13:04.622396 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.622371    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"]
Aug 05 17:13:04.623265 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.623244    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"]
Aug 05 17:13:04.643007 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.642984    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mj85f"]
Aug 05 17:13:04.646853 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.646834    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4jkwc"]
Aug 05 17:13:04.647060 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.647036    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:04.648457 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.648438    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"]
Aug 05 17:13:04.648557 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.648544    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4jkwc"
Aug 05 17:13:04.649884 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.649865    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9hnrl\""
Aug 05 17:13:04.650156 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.650069    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 05 17:13:04.650699 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.650680    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 05 17:13:04.651031 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.651011    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4qzqd\""
Aug 05 17:13:04.651310 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.651294    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 05 17:13:04.651360 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.651317    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 05 17:13:04.651831 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.651812    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 05 17:13:04.657618 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.657568    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4jkwc"]
Aug 05 17:13:04.658527 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.658507    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mj85f"]
Aug 05 17:13:04.732922 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.732892    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.732922 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.732925    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.733193 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.732952    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-bound-sa-token\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.733193 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733020    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-ca\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.733193 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733066    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84gfl\" (UniqueName: \"kubernetes.io/projected/8feeab71-5d14-4b7b-a0d5-54e9439b5078-kube-api-access-84gfl\") pod \"ingress-canary-4jkwc\" (UID: \"8feeab71-5d14-4b7b-a0d5-54e9439b5078\") " pod="openshift-ingress-canary/ingress-canary-4jkwc"
Aug 05 17:13:04.733193 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733112    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-hub\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.733193 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733150    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.733193 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733177    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a05c3353-9f73-4a22-8ab5-6ed05e46277c-tmp\") pod \"klusterlet-addon-workmgr-6bfb6446d9-hkh28\" (UID: \"a05c3353-9f73-4a22-8ab5-6ed05e46277c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:13:04.733429 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/066a1a18-72fc-4047-a6c8-527dcbca50c0-tmp-dir\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:04.733429 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733222    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqqkq\" (UniqueName: \"kubernetes.io/projected/9a68b48b-c688-4944-b9ca-84e3aa5d83da-kube-api-access-nqqkq\") pod \"managed-serviceaccount-addon-agent-69cd655b84-n2pqv\" (UID: \"9a68b48b-c688-4944-b9ca-84e3aa5d83da\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv"
Aug 05 17:13:04.733429 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733268    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.733429 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733287    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f07541a-09e0-4b0d-804a-61698af7ece3-installation-pull-secrets\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.733429 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733315    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54mst\" (UniqueName: \"kubernetes.io/projected/a05c3353-9f73-4a22-8ab5-6ed05e46277c-kube-api-access-54mst\") pod \"klusterlet-addon-workmgr-6bfb6446d9-hkh28\" (UID: \"a05c3353-9f73-4a22-8ab5-6ed05e46277c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:13:04.733429 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a05c3353-9f73-4a22-8ab5-6ed05e46277c-klusterlet-config\") pod \"klusterlet-addon-workmgr-6bfb6446d9-hkh28\" (UID: \"a05c3353-9f73-4a22-8ab5-6ed05e46277c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:13:04.733429 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733419    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-826zj\" (UniqueName: \"kubernetes.io/projected/066a1a18-72fc-4047-a6c8-527dcbca50c0-kube-api-access-826zj\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:04.733706 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733445    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f07541a-09e0-4b0d-804a-61698af7ece3-image-registry-private-configuration\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.733706 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733465    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f07541a-09e0-4b0d-804a-61698af7ece3-ca-trust-extracted\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.733706 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733520    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8feeab71-5d14-4b7b-a0d5-54e9439b5078-cert\") pod \"ingress-canary-4jkwc\" (UID: \"8feeab71-5d14-4b7b-a0d5-54e9439b5078\") " pod="openshift-ingress-canary/ingress-canary-4jkwc"
Aug 05 17:13:04.733706 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733563    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f07541a-09e0-4b0d-804a-61698af7ece3-trusted-ca\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.733706 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733586    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/066a1a18-72fc-4047-a6c8-527dcbca50c0-config-volume\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:04.733706 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733606    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z9j8\" (UniqueName: \"kubernetes.io/projected/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-kube-api-access-6z9j8\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.733706 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733622    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-certificates\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.733706 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733646    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a68b48b-c688-4944-b9ca-84e3aa5d83da-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69cd655b84-n2pqv\" (UID: \"9a68b48b-c688-4944-b9ca-84e3aa5d83da\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv"
Aug 05 17:13:04.733706 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733670    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnnmp\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-kube-api-access-cnnmp\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.733706 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.733708    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:04.834732 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.834638    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.834732 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.834685    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f07541a-09e0-4b0d-804a-61698af7ece3-installation-pull-secrets\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.834732 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.834715    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54mst\" (UniqueName: \"kubernetes.io/projected/a05c3353-9f73-4a22-8ab5-6ed05e46277c-kube-api-access-54mst\") pod \"klusterlet-addon-workmgr-6bfb6446d9-hkh28\" (UID: \"a05c3353-9f73-4a22-8ab5-6ed05e46277c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:13:04.834992 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.834762    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a05c3353-9f73-4a22-8ab5-6ed05e46277c-klusterlet-config\") pod \"klusterlet-addon-workmgr-6bfb6446d9-hkh28\" (UID: \"a05c3353-9f73-4a22-8ab5-6ed05e46277c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:13:04.834992 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.834791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-826zj\" (UniqueName: \"kubernetes.io/projected/066a1a18-72fc-4047-a6c8-527dcbca50c0-kube-api-access-826zj\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:04.834992 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:04.834801    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:13:04.834992 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:04.834826    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74b9c8bfb8-gdpw5: secret "image-registry-tls" not found
Aug 05 17:13:04.834992 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:04.834915    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls podName:5f07541a-09e0-4b0d-804a-61698af7ece3 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:05.334890159 +0000 UTC m=+35.626744987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls") pod "image-registry-74b9c8bfb8-gdpw5" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3") : secret "image-registry-tls" not found
Aug 05 17:13:04.835234 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.834810    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f07541a-09e0-4b0d-804a-61698af7ece3-image-registry-private-configuration\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.835234 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835118    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f07541a-09e0-4b0d-804a-61698af7ece3-ca-trust-extracted\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.835234 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835153    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8feeab71-5d14-4b7b-a0d5-54e9439b5078-cert\") pod \"ingress-canary-4jkwc\" (UID: \"8feeab71-5d14-4b7b-a0d5-54e9439b5078\") " pod="openshift-ingress-canary/ingress-canary-4jkwc"
Aug 05 17:13:04.835234 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835183    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f07541a-09e0-4b0d-804a-61698af7ece3-trusted-ca\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.835234 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/066a1a18-72fc-4047-a6c8-527dcbca50c0-config-volume\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:04.835234 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835231    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z9j8\" (UniqueName: \"kubernetes.io/projected/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-kube-api-access-6z9j8\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.835474 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835251    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-certificates\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.835474 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835282    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a68b48b-c688-4944-b9ca-84e3aa5d83da-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69cd655b84-n2pqv\" (UID: \"9a68b48b-c688-4944-b9ca-84e3aa5d83da\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv"
Aug 05 17:13:04.835474 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835306    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnmp\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-kube-api-access-cnnmp\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.835474 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835331    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:04.835474 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835357    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.835474 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835379    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.835474 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835403    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-bound-sa-token\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.835474 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835439    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-ca\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.835474 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84gfl\" (UniqueName: \"kubernetes.io/projected/8feeab71-5d14-4b7b-a0d5-54e9439b5078-kube-api-access-84gfl\") pod \"ingress-canary-4jkwc\" (UID: \"8feeab71-5d14-4b7b-a0d5-54e9439b5078\") " pod="openshift-ingress-canary/ingress-canary-4jkwc"
Aug 05 17:13:04.835865 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835488    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-hub\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.835865 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835508    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.835865 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835528    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f07541a-09e0-4b0d-804a-61698af7ece3-ca-trust-extracted\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.835865 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835531    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a05c3353-9f73-4a22-8ab5-6ed05e46277c-tmp\") pod \"klusterlet-addon-workmgr-6bfb6446d9-hkh28\" (UID: \"a05c3353-9f73-4a22-8ab5-6ed05e46277c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:13:04.836835 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.835866    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a05c3353-9f73-4a22-8ab5-6ed05e46277c-tmp\") pod \"klusterlet-addon-workmgr-6bfb6446d9-hkh28\" (UID: \"a05c3353-9f73-4a22-8ab5-6ed05e46277c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:13:04.836835 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.836057    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f07541a-09e0-4b0d-804a-61698af7ece3-trusted-ca\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.836835 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.836595    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.836988 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.836876    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/066a1a18-72fc-4047-a6c8-527dcbca50c0-tmp-dir\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:04.836988 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.836914    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqqkq\" (UniqueName: \"kubernetes.io/projected/9a68b48b-c688-4944-b9ca-84e3aa5d83da-kube-api-access-nqqkq\") pod \"managed-serviceaccount-addon-agent-69cd655b84-n2pqv\" (UID: \"9a68b48b-c688-4944-b9ca-84e3aa5d83da\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv"
Aug 05 17:13:04.838332 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.837667    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/066a1a18-72fc-4047-a6c8-527dcbca50c0-config-volume\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:04.838332 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:04.837943    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:04.838332 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:04.838007    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls podName:066a1a18-72fc-4047-a6c8-527dcbca50c0 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:05.337988909 +0000 UTC m=+35.629843721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls") pod "dns-default-mj85f" (UID: "066a1a18-72fc-4047-a6c8-527dcbca50c0") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:04.838332 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.838277    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/066a1a18-72fc-4047-a6c8-527dcbca50c0-tmp-dir\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:04.839190 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.839168    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-certificates\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.841476 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.840895    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f07541a-09e0-4b0d-804a-61698af7ece3-image-registry-private-configuration\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.841476 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.840965    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-hub\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.841476 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.841169    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.841476 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.841437    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a05c3353-9f73-4a22-8ab5-6ed05e46277c-klusterlet-config\") pod \"klusterlet-addon-workmgr-6bfb6446d9-hkh28\" (UID: \"a05c3353-9f73-4a22-8ab5-6ed05e46277c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:13:04.841738 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.841476    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f07541a-09e0-4b0d-804a-61698af7ece3-installation-pull-secrets\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.842286 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.842266    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8feeab71-5d14-4b7b-a0d5-54e9439b5078-cert\") pod \"ingress-canary-4jkwc\" (UID: \"8feeab71-5d14-4b7b-a0d5-54e9439b5078\") " pod="openshift-ingress-canary/ingress-canary-4jkwc"
Aug 05 17:13:04.842393 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.842311    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.842699 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.842679    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-ca\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.843001 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.842981    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a68b48b-c688-4944-b9ca-84e3aa5d83da-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69cd655b84-n2pqv\" (UID: \"9a68b48b-c688-4944-b9ca-84e3aa5d83da\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv"
Aug 05 17:13:04.846526 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.846481    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54mst\" (UniqueName: \"kubernetes.io/projected/a05c3353-9f73-4a22-8ab5-6ed05e46277c-kube-api-access-54mst\") pod \"klusterlet-addon-workmgr-6bfb6446d9-hkh28\" (UID: \"a05c3353-9f73-4a22-8ab5-6ed05e46277c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:13:04.847273 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.847242    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-826zj\" (UniqueName: \"kubernetes.io/projected/066a1a18-72fc-4047-a6c8-527dcbca50c0-kube-api-access-826zj\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:04.849686 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.849657    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-bound-sa-token\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.849686 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.849689    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnmp\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-kube-api-access-cnnmp\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:04.849859 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.849828    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqqkq\" (UniqueName: \"kubernetes.io/projected/9a68b48b-c688-4944-b9ca-84e3aa5d83da-kube-api-access-nqqkq\") pod \"managed-serviceaccount-addon-agent-69cd655b84-n2pqv\" (UID: \"9a68b48b-c688-4944-b9ca-84e3aa5d83da\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv"
Aug 05 17:13:04.849919 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.849861    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z9j8\" (UniqueName: \"kubernetes.io/projected/4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32-kube-api-access-6z9j8\") pod \"cluster-proxy-proxy-agent-5d88777749-2wwpl\" (UID: \"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.851698 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.851677    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84gfl\" (UniqueName: \"kubernetes.io/projected/8feeab71-5d14-4b7b-a0d5-54e9439b5078-kube-api-access-84gfl\") pod \"ingress-canary-4jkwc\" (UID: \"8feeab71-5d14-4b7b-a0d5-54e9439b5078\") " pod="openshift-ingress-canary/ingress-canary-4jkwc"
Aug 05 17:13:04.913201 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.913171    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv"
Aug 05 17:13:04.935031 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.934990    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:13:04.943905 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.943877    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:13:04.966453 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:04.966428    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4jkwc"
Aug 05 17:13:05.188754 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:05.188671    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:13:05.188754 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:05.188713    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:13:05.191831 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:05.191805    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 05 17:13:05.192109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:05.192084    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 05 17:13:05.192289 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:05.192265    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wztxg\""
Aug 05 17:13:05.192365 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:05.192295    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 05 17:13:05.342021 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:05.341979    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:05.342218 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:05.342108    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:05.342218 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:05.342179    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:13:05.342218 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:05.342202    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74b9c8bfb8-gdpw5: secret "image-registry-tls" not found
Aug 05 17:13:05.342373 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:05.342236    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:05.342373 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:05.342290    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls podName:5f07541a-09e0-4b0d-804a-61698af7ece3 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:06.342266774 +0000 UTC m=+36.634121586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls") pod "image-registry-74b9c8bfb8-gdpw5" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3") : secret "image-registry-tls" not found
Aug 05 17:13:05.342373 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:05.342310    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls podName:066a1a18-72fc-4047-a6c8-527dcbca50c0 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:06.342300521 +0000 UTC m=+36.634155333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls") pod "dns-default-mj85f" (UID: "066a1a18-72fc-4047-a6c8-527dcbca50c0") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:05.687462 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:05.687362    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv"]
Aug 05 17:13:05.690802 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:05.689779    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"]
Aug 05 17:13:05.695068 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:13:05.695036    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a68b48b_c688_4944_b9ca_84e3aa5d83da.slice/crio-19b217a502922ad7a59743327bea78865fc4b7d6d90075fa040b2be3b6f01ae4 WatchSource:0}: Error finding container 19b217a502922ad7a59743327bea78865fc4b7d6d90075fa040b2be3b6f01ae4: Status 404 returned error can't find the container with id 19b217a502922ad7a59743327bea78865fc4b7d6d90075fa040b2be3b6f01ae4
Aug 05 17:13:05.698699 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:13:05.698674    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f3ac7b3_0ba7_4f3b_af1b_3b377507cc32.slice/crio-6a61f6933badb85d9d9800dd5a57fd1a342ba45725e123edbfa8879a1d4e7ada WatchSource:0}: Error finding container 6a61f6933badb85d9d9800dd5a57fd1a342ba45725e123edbfa8879a1d4e7ada: Status 404 returned error can't find the container with id 6a61f6933badb85d9d9800dd5a57fd1a342ba45725e123edbfa8879a1d4e7ada
Aug 05 17:13:05.708186 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:05.708163    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4jkwc"]
Aug 05 17:13:05.713500 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:13:05.713468    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8feeab71_5d14_4b7b_a0d5_54e9439b5078.slice/crio-f9fd964a120aba464f087b9722df10ef15cf4bd0177a280be026899164a71f11 WatchSource:0}: Error finding container f9fd964a120aba464f087b9722df10ef15cf4bd0177a280be026899164a71f11: Status 404 returned error can't find the container with id f9fd964a120aba464f087b9722df10ef15cf4bd0177a280be026899164a71f11
Aug 05 17:13:05.715881 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:05.715860    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"]
Aug 05 17:13:05.719979 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:13:05.719957    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05c3353_9f73_4a22_8ab5_6ed05e46277c.slice/crio-6f2f0e1a7d60983967354c532ffc1ed3151937507bc3626525aeed40ba4a3950 WatchSource:0}: Error finding container 6f2f0e1a7d60983967354c532ffc1ed3151937507bc3626525aeed40ba4a3950: Status 404 returned error can't find the container with id 6f2f0e1a7d60983967354c532ffc1ed3151937507bc3626525aeed40ba4a3950
Aug 05 17:13:06.188911 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:06.188828    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:13:06.193312 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:06.193291    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4hs5f\""
Aug 05 17:13:06.193504 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:06.193491    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 05 17:13:06.353369 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:06.353335    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:06.353554 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:06.353458    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:06.353554 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:06.353516    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:13:06.353554 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:06.353541    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74b9c8bfb8-gdpw5: secret "image-registry-tls" not found
Aug 05 17:13:06.353709 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:06.353580    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:06.353709 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:06.353618    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls podName:5f07541a-09e0-4b0d-804a-61698af7ece3 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:08.353594702 +0000 UTC m=+38.645449525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls") pod "image-registry-74b9c8bfb8-gdpw5" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3") : secret "image-registry-tls" not found
Aug 05 17:13:06.353709 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:06.353638    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls podName:066a1a18-72fc-4047-a6c8-527dcbca50c0 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:08.353628109 +0000 UTC m=+38.645482921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls") pod "dns-default-mj85f" (UID: "066a1a18-72fc-4047-a6c8-527dcbca50c0") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:06.456044 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:06.455958    2574 generic.go:356] "Generic (PLEG): container finished" podID="4a578ff9-e0d4-4386-a305-bcd010078bab" containerID="129450e1278075ef605a8bcf4f7e1ec06271d40616a3b7f9767518993718f343" exitCode=0
Aug 05 17:13:06.456223 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:06.456055    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6mhh" event={"ID":"4a578ff9-e0d4-4386-a305-bcd010078bab","Type":"ContainerDied","Data":"129450e1278075ef605a8bcf4f7e1ec06271d40616a3b7f9767518993718f343"}
Aug 05 17:13:06.458612 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:06.458584    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl" event={"ID":"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32","Type":"ContainerStarted","Data":"6a61f6933badb85d9d9800dd5a57fd1a342ba45725e123edbfa8879a1d4e7ada"}
Aug 05 17:13:06.461185 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:06.461155    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv" event={"ID":"9a68b48b-c688-4944-b9ca-84e3aa5d83da","Type":"ContainerStarted","Data":"19b217a502922ad7a59743327bea78865fc4b7d6d90075fa040b2be3b6f01ae4"}
Aug 05 17:13:06.465781 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:06.465758    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28" event={"ID":"a05c3353-9f73-4a22-8ab5-6ed05e46277c","Type":"ContainerStarted","Data":"6f2f0e1a7d60983967354c532ffc1ed3151937507bc3626525aeed40ba4a3950"}
Aug 05 17:13:06.468790 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:06.468752    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4jkwc" event={"ID":"8feeab71-5d14-4b7b-a0d5-54e9439b5078","Type":"ContainerStarted","Data":"f9fd964a120aba464f087b9722df10ef15cf4bd0177a280be026899164a71f11"}
Aug 05 17:13:07.482436 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:07.482394    2574 generic.go:356] "Generic (PLEG): container finished" podID="4a578ff9-e0d4-4386-a305-bcd010078bab" containerID="7bb3230c6e35edc59551ea38cdf4a63c2232c6e796bc02aad9841844d2b8c8b3" exitCode=0
Aug 05 17:13:07.482934 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:07.482467    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6mhh" event={"ID":"4a578ff9-e0d4-4386-a305-bcd010078bab","Type":"ContainerDied","Data":"7bb3230c6e35edc59551ea38cdf4a63c2232c6e796bc02aad9841844d2b8c8b3"}
Aug 05 17:13:08.375207 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:08.375165    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:08.375396 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:08.375269    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:08.375396 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:08.375389    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:08.375610 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:08.375393    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:13:08.375610 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:08.375410    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74b9c8bfb8-gdpw5: secret "image-registry-tls" not found
Aug 05 17:13:08.375610 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:08.375450    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls podName:066a1a18-72fc-4047-a6c8-527dcbca50c0 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:12.375430562 +0000 UTC m=+42.667285385 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls") pod "dns-default-mj85f" (UID: "066a1a18-72fc-4047-a6c8-527dcbca50c0") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:08.375610 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:08.375468    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls podName:5f07541a-09e0-4b0d-804a-61698af7ece3 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:12.375459192 +0000 UTC m=+42.667314015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls") pod "image-registry-74b9c8bfb8-gdpw5" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3") : secret "image-registry-tls" not found
Aug 05 17:13:08.491152 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:08.491089    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6mhh" event={"ID":"4a578ff9-e0d4-4386-a305-bcd010078bab","Type":"ContainerStarted","Data":"1506f0067edc80789b8412b8727c13f1854aeff688aad4072505d26a18fdeaac"}
Aug 05 17:13:08.519284 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:08.519220    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n6mhh" podStartSLOduration=7.431893438 podStartE2EDuration="38.519197743s" totalImagesPullingTime="31.087304305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:08.518038258 +0000 UTC m=+38.809893115" watchObservedRunningTime="2026-08-05 17:13:08.519197743 +0000 UTC m=+38.811052557"
Aug 05 17:13:12.409954 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:12.409910    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:12.410456 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:12.410016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:12.410456 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:12.410119    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:13:12.410456 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:12.410146    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74b9c8bfb8-gdpw5: secret "image-registry-tls" not found
Aug 05 17:13:12.410456 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:12.410219    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls podName:5f07541a-09e0-4b0d-804a-61698af7ece3 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:20.410202198 +0000 UTC m=+50.702057007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls") pod "image-registry-74b9c8bfb8-gdpw5" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3") : secret "image-registry-tls" not found
Aug 05 17:13:12.410456 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:12.410147    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:12.410456 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:12.410253    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls podName:066a1a18-72fc-4047-a6c8-527dcbca50c0 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:20.410244589 +0000 UTC m=+50.702099400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls") pod "dns-default-mj85f" (UID: "066a1a18-72fc-4047-a6c8-527dcbca50c0") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:15.511059 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:15.511021    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv" event={"ID":"9a68b48b-c688-4944-b9ca-84e3aa5d83da","Type":"ContainerStarted","Data":"f23b6485b2a4dfc203c0bbcd53cd5b35795a7d4d93339cc7741c1fdd44ab6b66"}
Aug 05 17:13:15.512476 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:15.512455    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28" event={"ID":"a05c3353-9f73-4a22-8ab5-6ed05e46277c","Type":"ContainerStarted","Data":"1b7ace0be0e2fd6fbfde78cfc36f6119eecc92c127401d7aa6699e328993ab3f"}
Aug 05 17:13:15.512629 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:15.512538    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:13:15.513901 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:15.513872    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4jkwc" event={"ID":"8feeab71-5d14-4b7b-a0d5-54e9439b5078","Type":"ContainerStarted","Data":"c9ef1304bd89390141b99eebede4da04b1e657f251944c5fb81a727951078a85"}
Aug 05 17:13:15.514595 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:15.514576    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:13:15.516059 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:15.516037    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl" event={"ID":"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32","Type":"ContainerStarted","Data":"41e017937192704702d6a45370ca7ff7a2df5eda8f6c993736363e1b25c473ec"}
Aug 05 17:13:15.516157 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:15.516065    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl" event={"ID":"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32","Type":"ContainerStarted","Data":"568f71fa980fec1be0d1764b2af8d3b4b3ac807f398bf8cf2479db240ef9d1b5"}
Aug 05 17:13:15.516157 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:15.516095    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl" event={"ID":"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32","Type":"ContainerStarted","Data":"b99327326f5d07c5dcffac4c5ab67e0b2d9db39cc5c7268da6297e3bcf7ad64e"}
Aug 05 17:13:15.535512 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:15.535469    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv" podStartSLOduration=15.79883374 podStartE2EDuration="24.535457546s" totalImagesPullingTime="8.736623806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:15.53437836 +0000 UTC m=+45.826233191" watchObservedRunningTime="2026-08-05 17:13:15.535457546 +0000 UTC m=+45.827312389"
Aug 05 17:13:15.562537 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:15.562490    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl" podStartSLOduration=15.813764743 podStartE2EDuration="24.562474319s" totalImagesPullingTime="8.748709576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:15.561707516 +0000 UTC m=+45.853562349" watchObservedRunningTime="2026-08-05 17:13:15.562474319 +0000 UTC m=+45.854329149"
Aug 05 17:13:15.581215 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:15.581162    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28" podStartSLOduration=15.869344256 podStartE2EDuration="24.581145777s" totalImagesPullingTime="8.711801521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:15.580397486 +0000 UTC m=+45.872252316" watchObservedRunningTime="2026-08-05 17:13:15.581145777 +0000 UTC m=+45.873000608"
Aug 05 17:13:15.601920 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:15.601865    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4jkwc" podStartSLOduration=2.884467643 podStartE2EDuration="11.601849632s" totalImagesPullingTime="8.717381989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:15.601196178 +0000 UTC m=+45.893051025" watchObservedRunningTime="2026-08-05 17:13:15.601849632 +0000 UTC m=+45.893704441"
Aug 05 17:13:20.472567 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:20.472520    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:20.472980 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:20.472600    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:20.472980 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:20.472680    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:13:20.472980 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:20.472702    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74b9c8bfb8-gdpw5: secret "image-registry-tls" not found
Aug 05 17:13:20.472980 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:20.472752    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls podName:5f07541a-09e0-4b0d-804a-61698af7ece3 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:36.472736905 +0000 UTC m=+66.764591715 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls") pod "image-registry-74b9c8bfb8-gdpw5" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3") : secret "image-registry-tls" not found
Aug 05 17:13:20.472980 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:20.472687    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:20.472980 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:20.472819    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls podName:066a1a18-72fc-4047-a6c8-527dcbca50c0 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:36.472807968 +0000 UTC m=+66.764662777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls") pod "dns-default-mj85f" (UID: "066a1a18-72fc-4047-a6c8-527dcbca50c0") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:30.449912 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:30.449883    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xrhbh"
Aug 05 17:13:34.980718 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:34.980664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs\") pod \"network-metrics-daemon-xwnbn\" (UID: \"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660\") " pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:13:34.980718 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:34.980734    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:13:34.983629 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:34.983608    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 05 17:13:34.984435 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:34.984421    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 05 17:13:34.991203 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:34.991185    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Aug 05 17:13:34.991262 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:34.991251    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs podName:a7531d8b-5f4c-43e0-bc2f-d9e2a415f660 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:38.991227246 +0000 UTC m=+129.283082056 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs") pod "network-metrics-daemon-xwnbn" (UID: "a7531d8b-5f4c-43e0-bc2f-d9e2a415f660") : secret "metrics-daemon-secret" not found
Aug 05 17:13:34.994965 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:34.994928    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/057a0fc7-bbc2-47e0-9b9f-d4e292b354f1-original-pull-secret\") pod \"global-pull-secret-syncer-w89xx\" (UID: \"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1\") " pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:13:35.081707 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:35.081655    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nn8v\" (UniqueName: \"kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v\") pod \"network-check-target-44mzx\" (UID: \"1fc45fe8-5a2b-4e42-bf7e-938c50805465\") " pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:13:35.084716 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:35.084697    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 05 17:13:35.094617 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:35.094593    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 05 17:13:35.105261 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:35.105240    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nn8v\" (UniqueName: \"kubernetes.io/projected/1fc45fe8-5a2b-4e42-bf7e-938c50805465-kube-api-access-8nn8v\") pod \"network-check-target-44mzx\" (UID: \"1fc45fe8-5a2b-4e42-bf7e-938c50805465\") " pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:13:35.202586 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:35.202557    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wztxg\""
Aug 05 17:13:35.205695 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:35.205679    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w89xx"
Aug 05 17:13:35.210417 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:35.210393    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:13:35.343487 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:35.343458    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-44mzx"]
Aug 05 17:13:35.346047 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:13:35.346019    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc45fe8_5a2b_4e42_bf7e_938c50805465.slice/crio-429401b0a1bf4eb758beaf1f4e95be0915c3d284c61ec394742c8783f2d8b5ea WatchSource:0}: Error finding container 429401b0a1bf4eb758beaf1f4e95be0915c3d284c61ec394742c8783f2d8b5ea: Status 404 returned error can't find the container with id 429401b0a1bf4eb758beaf1f4e95be0915c3d284c61ec394742c8783f2d8b5ea
Aug 05 17:13:35.362621 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:35.362591    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w89xx"]
Aug 05 17:13:35.366242 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:13:35.366214    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057a0fc7_bbc2_47e0_9b9f_d4e292b354f1.slice/crio-0c840288ad5e8088ba378d37da844c1071171166e9b455ecde8e60def83fbe42 WatchSource:0}: Error finding container 0c840288ad5e8088ba378d37da844c1071171166e9b455ecde8e60def83fbe42: Status 404 returned error can't find the container with id 0c840288ad5e8088ba378d37da844c1071171166e9b455ecde8e60def83fbe42
Aug 05 17:13:35.573493 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:35.573404    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w89xx" event={"ID":"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1","Type":"ContainerStarted","Data":"0c840288ad5e8088ba378d37da844c1071171166e9b455ecde8e60def83fbe42"}
Aug 05 17:13:35.574586 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:35.574562    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-44mzx" event={"ID":"1fc45fe8-5a2b-4e42-bf7e-938c50805465","Type":"ContainerStarted","Data":"429401b0a1bf4eb758beaf1f4e95be0915c3d284c61ec394742c8783f2d8b5ea"}
Aug 05 17:13:36.493387 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:36.493344    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:13:36.493867 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:36.493441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:13:36.493867 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:36.493593    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:36.493867 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:36.493649    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls podName:066a1a18-72fc-4047-a6c8-527dcbca50c0 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:08.493634103 +0000 UTC m=+98.785488912 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls") pod "dns-default-mj85f" (UID: "066a1a18-72fc-4047-a6c8-527dcbca50c0") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:36.494123 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:36.494040    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:13:36.494123 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:36.494057    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74b9c8bfb8-gdpw5: secret "image-registry-tls" not found
Aug 05 17:13:36.494123 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:13:36.494124    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls podName:5f07541a-09e0-4b0d-804a-61698af7ece3 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:08.494107155 +0000 UTC m=+98.785961970 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls") pod "image-registry-74b9c8bfb8-gdpw5" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3") : secret "image-registry-tls" not found
Aug 05 17:13:40.593268 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:40.593222    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w89xx" event={"ID":"057a0fc7-bbc2-47e0-9b9f-d4e292b354f1","Type":"ContainerStarted","Data":"fe5aa0c372d16cddeb10c684cd93dc84afcb5efe4496b7a5f562b4a1f54e57fe"}
Aug 05 17:13:40.594616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:40.594590    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-44mzx" event={"ID":"1fc45fe8-5a2b-4e42-bf7e-938c50805465","Type":"ContainerStarted","Data":"2c4d04f28ffb5be0022b7c3768761f620ba75e1e72daa9e346023f0008ea05c8"}
Aug 05 17:13:40.594741 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:40.594651    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:13:40.610968 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:40.610913    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w89xx" podStartSLOduration=66.326799478 podStartE2EDuration="1m10.610895814s" totalImagesPullingTime="4.284096336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:40.610792267 +0000 UTC m=+70.902647099" watchObservedRunningTime="2026-08-05 17:13:40.610895814 +0000 UTC m=+70.902750648"
Aug 05 17:13:40.629745 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:13:40.629699    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-44mzx" podStartSLOduration=66.329327783 podStartE2EDuration="1m10.629684381s" totalImagesPullingTime="4.300356598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:40.629279431 +0000 UTC m=+70.921134262" watchObservedRunningTime="2026-08-05 17:13:40.629684381 +0000 UTC m=+70.921539211"
Aug 05 17:14:08.535068 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:08.534936    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:14:08.535068 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:08.534994    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:14:08.535605 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:08.535127    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:14:08.535605 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:08.535139    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74b9c8bfb8-gdpw5: secret "image-registry-tls" not found
Aug 05 17:14:08.535605 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:08.535194    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls podName:5f07541a-09e0-4b0d-804a-61698af7ece3 nodeName:}" failed. No retries permitted until 2026-08-05 17:15:12.535180865 +0000 UTC m=+162.827035687 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls") pod "image-registry-74b9c8bfb8-gdpw5" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3") : secret "image-registry-tls" not found
Aug 05 17:14:08.535605 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:08.535127    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:14:08.535605 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:08.535274    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls podName:066a1a18-72fc-4047-a6c8-527dcbca50c0 nodeName:}" failed. No retries permitted until 2026-08-05 17:15:12.535257934 +0000 UTC m=+162.827112744 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls") pod "dns-default-mj85f" (UID: "066a1a18-72fc-4047-a6c8-527dcbca50c0") : secret "dns-default-metrics-tls" not found
Aug 05 17:14:11.599642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:11.599609    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-44mzx"
Aug 05 17:14:27.410206 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.410173    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5ldls"]
Aug 05 17:14:27.412228 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.412206    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:27.415426 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.415384    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 05 17:14:27.415545 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.415478    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 05 17:14:27.415816 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.415799    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 05 17:14:27.416779 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.416754    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 05 17:14:27.416904 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.416758    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h9fhk\""
Aug 05 17:14:27.432466 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.432438    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5ldls"]
Aug 05 17:14:27.571231 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.571197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-data-volume\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:27.571231 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.571236    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:27.571459 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.571260    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:27.571459 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.571351    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-crio-socket\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:27.571459 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.571388    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nmx9\" (UniqueName: \"kubernetes.io/projected/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-api-access-5nmx9\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:27.672105 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.671999    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-data-volume\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:27.672105 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.672041    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:27.672105 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.672063    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:27.672319 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.672158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-crio-socket\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:27.672319 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:27.672164    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 05 17:14:27.672319 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.672244    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-crio-socket\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:27.672319 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:27.672268    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm podName:f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:28.172248661 +0000 UTC m=+118.464103483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5ldls" (UID: "f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7") : configmap "kube-rbac-proxy" not found
Aug 05 17:14:27.672319 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:27.672289    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:27.672319 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.672295    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nmx9\" (UniqueName: \"kubernetes.io/projected/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-api-access-5nmx9\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:27.672559 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:27.672363    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls podName:f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:28.17234486 +0000 UTC m=+118.464199682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5ldls" (UID: "f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:27.672559 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.672389    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-data-volume\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:27.686941 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:27.686912    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nmx9\" (UniqueName: \"kubernetes.io/projected/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-api-access-5nmx9\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:28.175440 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:28.175398    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:28.175440 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:28.175445    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:28.175712 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:28.175528    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 05 17:14:28.175712 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:28.175584    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:28.175712 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:28.175597    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm podName:f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:29.175579921 +0000 UTC m=+119.467434730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5ldls" (UID: "f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7") : configmap "kube-rbac-proxy" not found
Aug 05 17:14:28.175712 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:28.175670    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls podName:f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:29.175653577 +0000 UTC m=+119.467508386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5ldls" (UID: "f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:29.184488 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:29.184431    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:29.184488 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:29.184489    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:29.184922 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:29.184573    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 05 17:14:29.184922 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:29.184647    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm podName:f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:31.184631901 +0000 UTC m=+121.476486722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5ldls" (UID: "f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7") : configmap "kube-rbac-proxy" not found
Aug 05 17:14:29.184922 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:29.184578    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:29.184922 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:29.184684    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls podName:f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:31.184674793 +0000 UTC m=+121.476529602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5ldls" (UID: "f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:29.329863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:29.329838    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kgdd2_fc979956-0bb5-4600-81bb-19a5521be56d/dns-node-resolver/0.log"
Aug 05 17:14:30.329108 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:30.329035    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hsn68_ef418180-36b0-450a-8ac5-8518f605f717/node-ca/0.log"
Aug 05 17:14:30.727468 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:30.727382    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4jkwc_8feeab71-5d14-4b7b-a0d5-54e9439b5078/serve-healthcheck-canary/0.log"
Aug 05 17:14:31.200302 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:31.200261    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:31.200302 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:31.200301    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:31.200521 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:31.200389    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 05 17:14:31.200521 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:31.200445    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:31.200521 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:31.200479    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm podName:f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:35.200463136 +0000 UTC m=+125.492317950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5ldls" (UID: "f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7") : configmap "kube-rbac-proxy" not found
Aug 05 17:14:31.200521 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:31.200496    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls podName:f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:35.200488647 +0000 UTC m=+125.492343457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5ldls" (UID: "f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:35.234393 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:35.234351    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:35.234393 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:35.234393    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:35.234919 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:35.234479    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 05 17:14:35.234919 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:35.234505    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:35.234919 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:35.234554    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls podName:f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:43.234540374 +0000 UTC m=+133.526395184 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5ldls" (UID: "f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:35.234919 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:35.234567    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm podName:f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:43.234560303 +0000 UTC m=+133.526415111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5ldls" (UID: "f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7") : configmap "kube-rbac-proxy" not found
Aug 05 17:14:39.064045 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:39.063983    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs\") pod \"network-metrics-daemon-xwnbn\" (UID: \"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660\") " pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:14:39.064492 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:39.064160    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Aug 05 17:14:39.064492 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:39.064224    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs podName:a7531d8b-5f4c-43e0-bc2f-d9e2a415f660 nodeName:}" failed. No retries permitted until 2026-08-05 17:16:41.064206187 +0000 UTC m=+251.356061014 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs") pod "network-metrics-daemon-xwnbn" (UID: "a7531d8b-5f4c-43e0-bc2f-d9e2a415f660") : secret "metrics-daemon-secret" not found
Aug 05 17:14:43.294697 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:43.294655    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:43.294697 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:43.294695    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:43.295240 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:43.294826    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:43.295240 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:14:43.294902    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls podName:f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7 nodeName:}" failed. No retries permitted until 2026-08-05 17:14:59.294883261 +0000 UTC m=+149.586738075 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5ldls" (UID: "f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:14:43.295333 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:43.295313    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:59.318356 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:59.318297    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:59.320566 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:59.320540    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5ldls\" (UID: \"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7\") " pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:59.524055 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:59.524020    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h9fhk\""
Aug 05 17:14:59.532205 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:59.532183    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5ldls"
Aug 05 17:14:59.658212 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:59.658182    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5ldls"]
Aug 05 17:14:59.661241 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:14:59.661218    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d5390c_bc05_4dd2_8c64_dd73ad70b7d7.slice/crio-2d2d54cc25e5cba0e7f515f2276af4c5978f73c5987d5c71b928858370d2d7e2 WatchSource:0}: Error finding container 2d2d54cc25e5cba0e7f515f2276af4c5978f73c5987d5c71b928858370d2d7e2: Status 404 returned error can't find the container with id 2d2d54cc25e5cba0e7f515f2276af4c5978f73c5987d5c71b928858370d2d7e2
Aug 05 17:14:59.823824 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:59.823790    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5ldls" event={"ID":"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7","Type":"ContainerStarted","Data":"bead7e865bda98ec7094322624acdb1699465cf64d6989602ce4c2ca11f209f9"}
Aug 05 17:14:59.823987 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:14:59.823871    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5ldls" event={"ID":"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7","Type":"ContainerStarted","Data":"2d2d54cc25e5cba0e7f515f2276af4c5978f73c5987d5c71b928858370d2d7e2"}
Aug 05 17:15:00.828391 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:00.828354    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5ldls" event={"ID":"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7","Type":"ContainerStarted","Data":"fd53bac0f959de54fa01201c7d39194ccd9585ff82c62b8282648acc1417e1c9"}
Aug 05 17:15:02.836128 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:02.836092    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5ldls" event={"ID":"f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7","Type":"ContainerStarted","Data":"f147b147155248b024a274b51bf68d658e4797ab9f199e38dbb5891e645be50e"}
Aug 05 17:15:02.860735 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:02.860682    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5ldls" podStartSLOduration=33.619481977 podStartE2EDuration="35.860665347s" totalImagesPullingTime="2.24118337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:14:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:02.860032434 +0000 UTC m=+153.151887265" watchObservedRunningTime="2026-08-05 17:15:02.860665347 +0000 UTC m=+153.152520177"
Aug 05 17:15:07.649353 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:15:07.649306    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5" podUID="5f07541a-09e0-4b0d-804a-61698af7ece3"
Aug 05 17:15:07.660608 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:15:07.660573    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mj85f" podUID="066a1a18-72fc-4047-a6c8-527dcbca50c0"
Aug 05 17:15:07.849750 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:07.849718    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mj85f"
Aug 05 17:15:07.849951 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:07.849718    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:15:09.198230 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:15:09.198182    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xwnbn" podUID="a7531d8b-5f4c-43e0-bc2f-d9e2a415f660"
Aug 05 17:15:12.617495 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:12.617445    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:15:12.617909 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:12.617544    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:15:12.619849 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:12.619824    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/066a1a18-72fc-4047-a6c8-527dcbca50c0-metrics-tls\") pod \"dns-default-mj85f\" (UID: \"066a1a18-72fc-4047-a6c8-527dcbca50c0\") " pod="openshift-dns/dns-default-mj85f"
Aug 05 17:15:12.619952 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:12.619925    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls\") pod \"image-registry-74b9c8bfb8-gdpw5\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") " pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:15:12.656854 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:12.656825    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xrzw4\""
Aug 05 17:15:12.656854 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:12.656840    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9hnrl\""
Aug 05 17:15:12.661527 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:12.661506    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:15:12.661608 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:12.661592    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mj85f"
Aug 05 17:15:12.808377 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:12.808346    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mj85f"]
Aug 05 17:15:12.812474 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:15:12.812442    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066a1a18_72fc_4047_a6c8_527dcbca50c0.slice/crio-a6657228aeb689d161bc29bafe5ef9cce1fa1d66f442a14ae17c0973bcc38fa3 WatchSource:0}: Error finding container a6657228aeb689d161bc29bafe5ef9cce1fa1d66f442a14ae17c0973bcc38fa3: Status 404 returned error can't find the container with id a6657228aeb689d161bc29bafe5ef9cce1fa1d66f442a14ae17c0973bcc38fa3
Aug 05 17:15:12.824759 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:12.824736    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"]
Aug 05 17:15:12.825240 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:15:12.825213    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f07541a_09e0_4b0d_804a_61698af7ece3.slice/crio-8539f340cb80880191944fa160ccef8ec8b76ef073c46ff285b8178043aedf67 WatchSource:0}: Error finding container 8539f340cb80880191944fa160ccef8ec8b76ef073c46ff285b8178043aedf67: Status 404 returned error can't find the container with id 8539f340cb80880191944fa160ccef8ec8b76ef073c46ff285b8178043aedf67
Aug 05 17:15:12.867056 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:12.867025    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5" event={"ID":"5f07541a-09e0-4b0d-804a-61698af7ece3","Type":"ContainerStarted","Data":"8539f340cb80880191944fa160ccef8ec8b76ef073c46ff285b8178043aedf67"}
Aug 05 17:15:12.868047 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:12.867998    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mj85f" event={"ID":"066a1a18-72fc-4047-a6c8-527dcbca50c0","Type":"ContainerStarted","Data":"a6657228aeb689d161bc29bafe5ef9cce1fa1d66f442a14ae17c0973bcc38fa3"}
Aug 05 17:15:13.873455 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:13.873417    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5" event={"ID":"5f07541a-09e0-4b0d-804a-61698af7ece3","Type":"ContainerStarted","Data":"ef86179ce6777f13ead1fe860535d7b8af9ad07f007b7c7eb40e6e514af2bc27"}
Aug 05 17:15:13.874007 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:13.873541    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:15:13.897197 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:13.897138    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5" podStartSLOduration=163.897121813 podStartE2EDuration="2m43.897121813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:13.896731592 +0000 UTC m=+164.188586424" watchObservedRunningTime="2026-08-05 17:15:13.897121813 +0000 UTC m=+164.188976644"
Aug 05 17:15:14.877622 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:14.877545    2574 generic.go:356] "Generic (PLEG): container finished" podID="a05c3353-9f73-4a22-8ab5-6ed05e46277c" containerID="1b7ace0be0e2fd6fbfde78cfc36f6119eecc92c127401d7aa6699e328993ab3f" exitCode=1
Aug 05 17:15:14.877991 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:14.877614    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28" event={"ID":"a05c3353-9f73-4a22-8ab5-6ed05e46277c","Type":"ContainerDied","Data":"1b7ace0be0e2fd6fbfde78cfc36f6119eecc92c127401d7aa6699e328993ab3f"}
Aug 05 17:15:14.877991 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:14.877950    2574 scope.go:117] "RemoveContainer" containerID="1b7ace0be0e2fd6fbfde78cfc36f6119eecc92c127401d7aa6699e328993ab3f"
Aug 05 17:15:14.879208 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:14.879153    2574 generic.go:356] "Generic (PLEG): container finished" podID="9a68b48b-c688-4944-b9ca-84e3aa5d83da" containerID="f23b6485b2a4dfc203c0bbcd53cd5b35795a7d4d93339cc7741c1fdd44ab6b66" exitCode=255
Aug 05 17:15:14.879259 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:14.879233    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv" event={"ID":"9a68b48b-c688-4944-b9ca-84e3aa5d83da","Type":"ContainerDied","Data":"f23b6485b2a4dfc203c0bbcd53cd5b35795a7d4d93339cc7741c1fdd44ab6b66"}
Aug 05 17:15:14.879584 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:14.879570    2574 scope.go:117] "RemoveContainer" containerID="f23b6485b2a4dfc203c0bbcd53cd5b35795a7d4d93339cc7741c1fdd44ab6b66"
Aug 05 17:15:14.913423 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:14.913397    2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv"
Aug 05 17:15:14.945067 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:14.944997    2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:15:15.365419 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.365385    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bw488"]
Aug 05 17:15:15.369477 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.369461    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.372770 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.372749    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 05 17:15:15.373459 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.373441    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-64j8g\""
Aug 05 17:15:15.373530 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.373468    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 05 17:15:15.373776 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.373760    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 05 17:15:15.375614 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.375433    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 05 17:15:15.375614 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.375510    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 05 17:15:15.375614 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.375523    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 05 17:15:15.439928 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.439891    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3eb12115-1963-4b6e-bcb5-02c3fe687086-sys\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.439928 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.439926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-accelerators-collector-config\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.440172 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.439954    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb12115-1963-4b6e-bcb5-02c3fe687086-metrics-client-ca\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.440172 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.439981    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3eb12115-1963-4b6e-bcb5-02c3fe687086-root\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.440172 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.440016    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.440172 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.440115    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-tls\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.440172 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.440158    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-wtmp\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.440342 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.440179    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4pnf\" (UniqueName: \"kubernetes.io/projected/3eb12115-1963-4b6e-bcb5-02c3fe687086-kube-api-access-x4pnf\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.440342 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.440205    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-textfile\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.512636 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.512597    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:15:15.540954 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.540923    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-tls\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.540954 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.540954    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-wtmp\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.541201 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.540972    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4pnf\" (UniqueName: \"kubernetes.io/projected/3eb12115-1963-4b6e-bcb5-02c3fe687086-kube-api-access-x4pnf\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.541201 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.541102    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-wtmp\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.541201 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:15:15.541110    2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Aug 05 17:15:15.541201 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:15:15.541175    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-tls podName:3eb12115-1963-4b6e-bcb5-02c3fe687086 nodeName:}" failed. No retries permitted until 2026-08-05 17:15:16.041156954 +0000 UTC m=+166.333011763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-tls") pod "node-exporter-bw488" (UID: "3eb12115-1963-4b6e-bcb5-02c3fe687086") : secret "node-exporter-tls" not found
Aug 05 17:15:15.541201 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.541109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-textfile\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.541383 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.541216    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3eb12115-1963-4b6e-bcb5-02c3fe687086-sys\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.541383 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.541243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-accelerators-collector-config\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.541383 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.541276    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb12115-1963-4b6e-bcb5-02c3fe687086-metrics-client-ca\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.541383 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.541301    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3eb12115-1963-4b6e-bcb5-02c3fe687086-sys\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.541383 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.541345    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3eb12115-1963-4b6e-bcb5-02c3fe687086-root\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.541383 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.541306    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3eb12115-1963-4b6e-bcb5-02c3fe687086-root\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.541632 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.541385    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-textfile\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.541632 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.541420    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.541839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.541820    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb12115-1963-4b6e-bcb5-02c3fe687086-metrics-client-ca\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.541876 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.541822    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-accelerators-collector-config\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.543573 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.543557    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.554672 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.554652    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4pnf\" (UniqueName: \"kubernetes.io/projected/3eb12115-1963-4b6e-bcb5-02c3fe687086-kube-api-access-x4pnf\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:15.886933 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.886895    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28" event={"ID":"a05c3353-9f73-4a22-8ab5-6ed05e46277c","Type":"ContainerStarted","Data":"7a5a10783994ec09b2bbdd2fa6090b4917f87c653d6a8f5bacb625c0c3327434"}
Aug 05 17:15:15.887422 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.887010    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:15:15.888847 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.888828    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6bfb6446d9-hkh28"
Aug 05 17:15:15.889020 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.889001    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mj85f" event={"ID":"066a1a18-72fc-4047-a6c8-527dcbca50c0","Type":"ContainerStarted","Data":"09e46af2c799102a3f2b373ba86b1185986df68db43b59deb02f9d34201dc511"}
Aug 05 17:15:15.889117 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.889040    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mj85f" event={"ID":"066a1a18-72fc-4047-a6c8-527dcbca50c0","Type":"ContainerStarted","Data":"d46abe69c14fb1a69c962ff475348e22d6570b08ad7441b90ab9d596da299693"}
Aug 05 17:15:15.889117 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.889069    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mj85f"
Aug 05 17:15:15.890659 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.890637    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69cd655b84-n2pqv" event={"ID":"9a68b48b-c688-4944-b9ca-84e3aa5d83da","Type":"ContainerStarted","Data":"65ff41c106ab75f7e73c95fe1eede0968b6189edb6485e6a638bc1dfd5b999a1"}
Aug 05 17:15:15.925481 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:15.925427    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mj85f" podStartSLOduration=129.922773356 podStartE2EDuration="2m11.925411775s" totalImagesPullingTime="2.002638419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:15.924811235 +0000 UTC m=+166.216666094" watchObservedRunningTime="2026-08-05 17:15:15.925411775 +0000 UTC m=+166.217266606"
Aug 05 17:15:16.048695 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:16.048656    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-tls\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:16.051024 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:16.051004    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3eb12115-1963-4b6e-bcb5-02c3fe687086-node-exporter-tls\") pod \"node-exporter-bw488\" (UID: \"3eb12115-1963-4b6e-bcb5-02c3fe687086\") " pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:16.278678 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:16.278625    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bw488"
Aug 05 17:15:16.292407 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:15:16.292368    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb12115_1963_4b6e_bcb5_02c3fe687086.slice/crio-7beac169c4027e34374181fa9f22c68428cf4075b5474d9e31398a7b494428e4 WatchSource:0}: Error finding container 7beac169c4027e34374181fa9f22c68428cf4075b5474d9e31398a7b494428e4: Status 404 returned error can't find the container with id 7beac169c4027e34374181fa9f22c68428cf4075b5474d9e31398a7b494428e4
Aug 05 17:15:16.896024 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:16.895984    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bw488" event={"ID":"3eb12115-1963-4b6e-bcb5-02c3fe687086","Type":"ContainerStarted","Data":"7beac169c4027e34374181fa9f22c68428cf4075b5474d9e31398a7b494428e4"}
Aug 05 17:15:17.900225 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:17.900188    2574 generic.go:356] "Generic (PLEG): container finished" podID="3eb12115-1963-4b6e-bcb5-02c3fe687086" containerID="c8ef8dc31b0b23c8b67793c9f5cd419ffb719026e25b4ceeec643076fa5c071c" exitCode=0
Aug 05 17:15:17.900716 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:17.900276    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bw488" event={"ID":"3eb12115-1963-4b6e-bcb5-02c3fe687086","Type":"ContainerDied","Data":"c8ef8dc31b0b23c8b67793c9f5cd419ffb719026e25b4ceeec643076fa5c071c"}
Aug 05 17:15:18.905297 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:18.905260    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bw488" event={"ID":"3eb12115-1963-4b6e-bcb5-02c3fe687086","Type":"ContainerStarted","Data":"9500f58b47d4b3318544733a30dfe376d4422a25e4e08511041b8dd2ad1cdf40"}
Aug 05 17:15:18.905729 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:18.905315    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bw488" event={"ID":"3eb12115-1963-4b6e-bcb5-02c3fe687086","Type":"ContainerStarted","Data":"c7a24047cf43e2c4492a62ee776d5f9319532a25a7e66250ef0a5bc54cf89394"}
Aug 05 17:15:18.927470 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:18.927407    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bw488" podStartSLOduration=2.9078388889999998 podStartE2EDuration="3.927384264s" totalImagesPullingTime="955.147457ms" totalInitContainerRuntime="64.397918ms" isStatefulPod=true podCreationTimestamp="2026-08-05 17:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:18.926970529 +0000 UTC m=+169.218825360" watchObservedRunningTime="2026-08-05 17:15:18.927384264 +0000 UTC m=+169.219239095"
Aug 05 17:15:22.188993 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:22.188889    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:15:25.897821 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:25.897787    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mj85f"
Aug 05 17:15:27.711585 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:27.711550    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"]
Aug 05 17:15:27.716199 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:27.716162    2574 patch_prober.go:28] interesting pod/image-registry-74b9c8bfb8-gdpw5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Aug 05 17:15:27.716335 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:27.716225    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5" podUID="5f07541a-09e0-4b0d-804a-61698af7ece3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:15:37.716527 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:37.716495    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:15:52.730804 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:52.730735    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5" podUID="5f07541a-09e0-4b0d-804a-61698af7ece3" containerName="registry" containerID="cri-o://ef86179ce6777f13ead1fe860535d7b8af9ad07f007b7c7eb40e6e514af2bc27" gracePeriod=30
Aug 05 17:15:52.967309 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:52.967284    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:15:53.013830 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.013800    2574 generic.go:356] "Generic (PLEG): container finished" podID="5f07541a-09e0-4b0d-804a-61698af7ece3" containerID="ef86179ce6777f13ead1fe860535d7b8af9ad07f007b7c7eb40e6e514af2bc27" exitCode=0
Aug 05 17:15:53.014012 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.013846    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"
Aug 05 17:15:53.014012 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.013852    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5" event={"ID":"5f07541a-09e0-4b0d-804a-61698af7ece3","Type":"ContainerDied","Data":"ef86179ce6777f13ead1fe860535d7b8af9ad07f007b7c7eb40e6e514af2bc27"}
Aug 05 17:15:53.014012 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.013893    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74b9c8bfb8-gdpw5" event={"ID":"5f07541a-09e0-4b0d-804a-61698af7ece3","Type":"ContainerDied","Data":"8539f340cb80880191944fa160ccef8ec8b76ef073c46ff285b8178043aedf67"}
Aug 05 17:15:53.014012 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.013912    2574 scope.go:117] "RemoveContainer" containerID="ef86179ce6777f13ead1fe860535d7b8af9ad07f007b7c7eb40e6e514af2bc27"
Aug 05 17:15:53.021629 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.021613    2574 scope.go:117] "RemoveContainer" containerID="ef86179ce6777f13ead1fe860535d7b8af9ad07f007b7c7eb40e6e514af2bc27"
Aug 05 17:15:53.021906 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:15:53.021875    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef86179ce6777f13ead1fe860535d7b8af9ad07f007b7c7eb40e6e514af2bc27\": container with ID starting with ef86179ce6777f13ead1fe860535d7b8af9ad07f007b7c7eb40e6e514af2bc27 not found: ID does not exist" containerID="ef86179ce6777f13ead1fe860535d7b8af9ad07f007b7c7eb40e6e514af2bc27"
Aug 05 17:15:53.021984 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.021909    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef86179ce6777f13ead1fe860535d7b8af9ad07f007b7c7eb40e6e514af2bc27"} err="failed to get container status \"ef86179ce6777f13ead1fe860535d7b8af9ad07f007b7c7eb40e6e514af2bc27\": rpc error: code = NotFound desc = could not find container \"ef86179ce6777f13ead1fe860535d7b8af9ad07f007b7c7eb40e6e514af2bc27\": container with ID starting with ef86179ce6777f13ead1fe860535d7b8af9ad07f007b7c7eb40e6e514af2bc27 not found: ID does not exist"
Aug 05 17:15:53.135556 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.135518    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-bound-sa-token\") pod \"5f07541a-09e0-4b0d-804a-61698af7ece3\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") "
Aug 05 17:15:53.135556 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.135554    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls\") pod \"5f07541a-09e0-4b0d-804a-61698af7ece3\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") "
Aug 05 17:15:53.135800 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.135573    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f07541a-09e0-4b0d-804a-61698af7ece3-trusted-ca\") pod \"5f07541a-09e0-4b0d-804a-61698af7ece3\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") "
Aug 05 17:15:53.135800 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.135597    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f07541a-09e0-4b0d-804a-61698af7ece3-image-registry-private-configuration\") pod \"5f07541a-09e0-4b0d-804a-61698af7ece3\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") "
Aug 05 17:15:53.135800 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.135644    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f07541a-09e0-4b0d-804a-61698af7ece3-ca-trust-extracted\") pod \"5f07541a-09e0-4b0d-804a-61698af7ece3\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") "
Aug 05 17:15:53.135800 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.135666    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnnmp\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-kube-api-access-cnnmp\") pod \"5f07541a-09e0-4b0d-804a-61698af7ece3\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") "
Aug 05 17:15:53.135800 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.135682    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-certificates\") pod \"5f07541a-09e0-4b0d-804a-61698af7ece3\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") "
Aug 05 17:15:53.135800 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.135714    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f07541a-09e0-4b0d-804a-61698af7ece3-installation-pull-secrets\") pod \"5f07541a-09e0-4b0d-804a-61698af7ece3\" (UID: \"5f07541a-09e0-4b0d-804a-61698af7ece3\") "
Aug 05 17:15:53.136403 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.136321    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5f07541a-09e0-4b0d-804a-61698af7ece3" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:15:53.136403 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.136369    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f07541a-09e0-4b0d-804a-61698af7ece3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5f07541a-09e0-4b0d-804a-61698af7ece3" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:15:53.138110 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.138044    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f07541a-09e0-4b0d-804a-61698af7ece3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5f07541a-09e0-4b0d-804a-61698af7ece3" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:15:53.138204 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.138188    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5f07541a-09e0-4b0d-804a-61698af7ece3" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:15:53.138263 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.138194    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5f07541a-09e0-4b0d-804a-61698af7ece3" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:15:53.138365 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.138345    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-kube-api-access-cnnmp" (OuterVolumeSpecName: "kube-api-access-cnnmp") pod "5f07541a-09e0-4b0d-804a-61698af7ece3" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3"). InnerVolumeSpecName "kube-api-access-cnnmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:15:53.138547 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.138525    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f07541a-09e0-4b0d-804a-61698af7ece3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5f07541a-09e0-4b0d-804a-61698af7ece3" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:15:53.144541 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.144509    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f07541a-09e0-4b0d-804a-61698af7ece3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5f07541a-09e0-4b0d-804a-61698af7ece3" (UID: "5f07541a-09e0-4b0d-804a-61698af7ece3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:15:53.237087 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.237047    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f07541a-09e0-4b0d-804a-61698af7ece3-trusted-ca\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:15:53.237272 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.237110    2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f07541a-09e0-4b0d-804a-61698af7ece3-image-registry-private-configuration\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:15:53.237272 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.237126    2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f07541a-09e0-4b0d-804a-61698af7ece3-ca-trust-extracted\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:15:53.237272 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.237138    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnnmp\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-kube-api-access-cnnmp\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:15:53.237272 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.237152    2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-certificates\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:15:53.237272 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.237164    2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f07541a-09e0-4b0d-804a-61698af7ece3-installation-pull-secrets\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:15:53.237272 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.237177    2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-bound-sa-token\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:15:53.237272 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.237189    2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f07541a-09e0-4b0d-804a-61698af7ece3-registry-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:15:53.337223 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.337189    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"]
Aug 05 17:15:53.344758 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:53.344723    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74b9c8bfb8-gdpw5"]
Aug 05 17:15:54.192640 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:54.192597    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f07541a-09e0-4b0d-804a-61698af7ece3" path="/var/lib/kubelet/pods/5f07541a-09e0-4b0d-804a-61698af7ece3/volumes"
Aug 05 17:15:54.936668 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:15:54.936629    2574 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl" podUID="4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Aug 05 17:16:04.937005 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:04.936959    2574 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl" podUID="4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Aug 05 17:16:14.936584 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:14.936543    2574 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl" podUID="4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Aug 05 17:16:14.936949 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:14.936618    2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl"
Aug 05 17:16:14.937202 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:14.937183    2574 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"41e017937192704702d6a45370ca7ff7a2df5eda8f6c993736363e1b25c473ec"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Aug 05 17:16:14.937236 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:14.937225    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl" podUID="4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32" containerName="service-proxy" containerID="cri-o://41e017937192704702d6a45370ca7ff7a2df5eda8f6c993736363e1b25c473ec" gracePeriod=30
Aug 05 17:16:15.091998 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:15.091968    2574 generic.go:356] "Generic (PLEG): container finished" podID="4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32" containerID="41e017937192704702d6a45370ca7ff7a2df5eda8f6c993736363e1b25c473ec" exitCode=2
Aug 05 17:16:15.092175 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:15.092005    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl" event={"ID":"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32","Type":"ContainerDied","Data":"41e017937192704702d6a45370ca7ff7a2df5eda8f6c993736363e1b25c473ec"}
Aug 05 17:16:16.099523 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:16.099481    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d88777749-2wwpl" event={"ID":"4f3ac7b3-0ba7-4f3b-af1b-3b377507cc32","Type":"ContainerStarted","Data":"76564a1c4ba43d3e280d83a52ec7a0503fcc26163eef0ae113ab7b7b0a4faf1b"}
Aug 05 17:16:41.081976 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:41.081925    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs\") pod \"network-metrics-daemon-xwnbn\" (UID: \"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660\") " pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:16:41.084252 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:41.084231    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7531d8b-5f4c-43e0-bc2f-d9e2a415f660-metrics-certs\") pod \"network-metrics-daemon-xwnbn\" (UID: \"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660\") " pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:16:41.093257 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:41.093235    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4hs5f\""
Aug 05 17:16:41.100518 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:41.100503    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xwnbn"
Aug 05 17:16:41.232154 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:41.232123    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xwnbn"]
Aug 05 17:16:41.235393 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:16:41.235364    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7531d8b_5f4c_43e0_bc2f_d9e2a415f660.slice/crio-51f35a5189b4baf2b830fb4768452f6a7428557d3cf7ecaf0d4155a4fdc7772d WatchSource:0}: Error finding container 51f35a5189b4baf2b830fb4768452f6a7428557d3cf7ecaf0d4155a4fdc7772d: Status 404 returned error can't find the container with id 51f35a5189b4baf2b830fb4768452f6a7428557d3cf7ecaf0d4155a4fdc7772d
Aug 05 17:16:42.183000 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:42.182933    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwnbn" event={"ID":"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660","Type":"ContainerStarted","Data":"51f35a5189b4baf2b830fb4768452f6a7428557d3cf7ecaf0d4155a4fdc7772d"}
Aug 05 17:16:43.188170 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:43.188132    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwnbn" event={"ID":"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660","Type":"ContainerStarted","Data":"2c5ed1156e9b8f128ad3c1691540caba1a841872ff110e87952a8db92345031f"}
Aug 05 17:16:43.188581 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:43.188222    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xwnbn" event={"ID":"a7531d8b-5f4c-43e0-bc2f-d9e2a415f660","Type":"ContainerStarted","Data":"8b9db60115f5b3da6fb37de26beb1490ae4d5afb5e7ac1e4004f2c2334e6c9d9"}
Aug 05 17:16:43.246136 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:16:43.246060    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xwnbn" podStartSLOduration=252.301036834 podStartE2EDuration="4m13.246041652s" totalImagesPullingTime="945.004818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:16:43.245032391 +0000 UTC m=+253.536887234" watchObservedRunningTime="2026-08-05 17:16:43.246041652 +0000 UTC m=+253.537896482"
Aug 05 17:17:30.076281 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:17:30.076251    2574 kubelet.go:1648] "Image garbage collection succeeded"
Aug 05 17:21:46.615522 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.615482    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"]
Aug 05 17:21:46.616047 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.615804    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f07541a-09e0-4b0d-804a-61698af7ece3" containerName="registry"
Aug 05 17:21:46.616047 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.615821    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f07541a-09e0-4b0d-804a-61698af7ece3" containerName="registry"
Aug 05 17:21:46.616047 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.615882    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f07541a-09e0-4b0d-804a-61698af7ece3" containerName="registry"
Aug 05 17:21:46.618782 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.618762    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"
Aug 05 17:21:46.622069 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.622048    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Aug 05 17:21:46.622210 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.622173    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Aug 05 17:21:46.622276 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.622250    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Aug 05 17:21:46.623209 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.623190    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-2n5w9\""
Aug 05 17:21:46.640665 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.640634    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"]
Aug 05 17:21:46.668979 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.668944    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-2b982"]
Aug 05 17:21:46.671857 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.671839    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2b982"
Aug 05 17:21:46.676911 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.676891    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Aug 05 17:21:46.677050 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.676958    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-qdf27\""
Aug 05 17:21:46.684158 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.684131    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2b982"]
Aug 05 17:21:46.800297 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.800255    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7ce1cc18-c8e3-40a3-a052-77442f8ebfe2-data\") pod \"seaweedfs-64568bcd49-2b982\" (UID: \"7ce1cc18-c8e3-40a3-a052-77442f8ebfe2\") " pod="kserve/seaweedfs-64568bcd49-2b982"
Aug 05 17:21:46.800297 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.800295    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsgnq\" (UniqueName: \"kubernetes.io/projected/7ce1cc18-c8e3-40a3-a052-77442f8ebfe2-kube-api-access-wsgnq\") pod \"seaweedfs-64568bcd49-2b982\" (UID: \"7ce1cc18-c8e3-40a3-a052-77442f8ebfe2\") " pod="kserve/seaweedfs-64568bcd49-2b982"
Aug 05 17:21:46.800516 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.800374    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/494262a5-2678-44d1-a10c-da1a7df87e1c-cert\") pod \"kserve-controller-manager-6f49dcd9ff-8g9t5\" (UID: \"494262a5-2678-44d1-a10c-da1a7df87e1c\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"
Aug 05 17:21:46.800516 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.800429    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvmbk\" (UniqueName: \"kubernetes.io/projected/494262a5-2678-44d1-a10c-da1a7df87e1c-kube-api-access-dvmbk\") pod \"kserve-controller-manager-6f49dcd9ff-8g9t5\" (UID: \"494262a5-2678-44d1-a10c-da1a7df87e1c\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"
Aug 05 17:21:46.901334 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.901256    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7ce1cc18-c8e3-40a3-a052-77442f8ebfe2-data\") pod \"seaweedfs-64568bcd49-2b982\" (UID: \"7ce1cc18-c8e3-40a3-a052-77442f8ebfe2\") " pod="kserve/seaweedfs-64568bcd49-2b982"
Aug 05 17:21:46.901334 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.901292    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsgnq\" (UniqueName: \"kubernetes.io/projected/7ce1cc18-c8e3-40a3-a052-77442f8ebfe2-kube-api-access-wsgnq\") pod \"seaweedfs-64568bcd49-2b982\" (UID: \"7ce1cc18-c8e3-40a3-a052-77442f8ebfe2\") " pod="kserve/seaweedfs-64568bcd49-2b982"
Aug 05 17:21:46.901334 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.901334    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/494262a5-2678-44d1-a10c-da1a7df87e1c-cert\") pod \"kserve-controller-manager-6f49dcd9ff-8g9t5\" (UID: \"494262a5-2678-44d1-a10c-da1a7df87e1c\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"
Aug 05 17:21:46.901544 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.901351    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvmbk\" (UniqueName: \"kubernetes.io/projected/494262a5-2678-44d1-a10c-da1a7df87e1c-kube-api-access-dvmbk\") pod \"kserve-controller-manager-6f49dcd9ff-8g9t5\" (UID: \"494262a5-2678-44d1-a10c-da1a7df87e1c\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"
Aug 05 17:21:46.901712 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.901676    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7ce1cc18-c8e3-40a3-a052-77442f8ebfe2-data\") pod \"seaweedfs-64568bcd49-2b982\" (UID: \"7ce1cc18-c8e3-40a3-a052-77442f8ebfe2\") " pod="kserve/seaweedfs-64568bcd49-2b982"
Aug 05 17:21:46.903783 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.903765    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/494262a5-2678-44d1-a10c-da1a7df87e1c-cert\") pod \"kserve-controller-manager-6f49dcd9ff-8g9t5\" (UID: \"494262a5-2678-44d1-a10c-da1a7df87e1c\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"
Aug 05 17:21:46.910639 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.910613    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvmbk\" (UniqueName: \"kubernetes.io/projected/494262a5-2678-44d1-a10c-da1a7df87e1c-kube-api-access-dvmbk\") pod \"kserve-controller-manager-6f49dcd9ff-8g9t5\" (UID: \"494262a5-2678-44d1-a10c-da1a7df87e1c\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"
Aug 05 17:21:46.910639 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.910629    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsgnq\" (UniqueName: \"kubernetes.io/projected/7ce1cc18-c8e3-40a3-a052-77442f8ebfe2-kube-api-access-wsgnq\") pod \"seaweedfs-64568bcd49-2b982\" (UID: \"7ce1cc18-c8e3-40a3-a052-77442f8ebfe2\") " pod="kserve/seaweedfs-64568bcd49-2b982"
Aug 05 17:21:46.928512 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.928480    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"
Aug 05 17:21:46.980248 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:46.980219    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2b982"
Aug 05 17:21:47.165960 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:47.165875    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2b982"]
Aug 05 17:21:47.168671 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:21:47.168647    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce1cc18_c8e3_40a3_a052_77442f8ebfe2.slice/crio-051ef3019ad062896e5004890843e51d12ab51c0fc7dc8c476ac3b27ef8c81d2 WatchSource:0}: Error finding container 051ef3019ad062896e5004890843e51d12ab51c0fc7dc8c476ac3b27ef8c81d2: Status 404 returned error can't find the container with id 051ef3019ad062896e5004890843e51d12ab51c0fc7dc8c476ac3b27ef8c81d2
Aug 05 17:21:47.169828 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:47.169814    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 05 17:21:47.262939 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:47.262903    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"]
Aug 05 17:21:47.267158 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:21:47.267127    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494262a5_2678_44d1_a10c_da1a7df87e1c.slice/crio-50cb6acb17c44ff514cc3303a932703652cb44245eb7a0e4b66f3553e5e3b30e WatchSource:0}: Error finding container 50cb6acb17c44ff514cc3303a932703652cb44245eb7a0e4b66f3553e5e3b30e: Status 404 returned error can't find the container with id 50cb6acb17c44ff514cc3303a932703652cb44245eb7a0e4b66f3553e5e3b30e
Aug 05 17:21:48.161794 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:48.161752    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5" event={"ID":"494262a5-2678-44d1-a10c-da1a7df87e1c","Type":"ContainerStarted","Data":"50cb6acb17c44ff514cc3303a932703652cb44245eb7a0e4b66f3553e5e3b30e"}
Aug 05 17:21:48.163733 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:48.163704    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2b982" event={"ID":"7ce1cc18-c8e3-40a3-a052-77442f8ebfe2","Type":"ContainerStarted","Data":"051ef3019ad062896e5004890843e51d12ab51c0fc7dc8c476ac3b27ef8c81d2"}
Aug 05 17:21:51.176736 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:51.176638    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5" event={"ID":"494262a5-2678-44d1-a10c-da1a7df87e1c","Type":"ContainerStarted","Data":"8edf3192f46baa01ffd003c186e718ed0a021cad76d15ae9de1bd39edf01bb4a"}
Aug 05 17:21:51.177224 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:51.176796    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"
Aug 05 17:21:51.178057 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:51.178030    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2b982" event={"ID":"7ce1cc18-c8e3-40a3-a052-77442f8ebfe2","Type":"ContainerStarted","Data":"1f4601c3a3d81583473a299d67e862aa2565a57c7c6ef24c774131c9b0342d25"}
Aug 05 17:21:51.178162 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:51.178135    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-2b982"
Aug 05 17:21:51.196418 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:51.196361    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5" podStartSLOduration=1.726087481 podStartE2EDuration="5.196344788s" totalImagesPullingTime="3.470257307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:51.195732417 +0000 UTC m=+561.487587246" watchObservedRunningTime="2026-08-05 17:21:51.196344788 +0000 UTC m=+561.488199620"
Aug 05 17:21:51.217218 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:51.217164    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-2b982" podStartSLOduration=1.5621546419999999 podStartE2EDuration="5.217146201s" totalImagesPullingTime="3.654991559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:51.216045571 +0000 UTC m=+561.507900406" watchObservedRunningTime="2026-08-05 17:21:51.217146201 +0000 UTC m=+561.509001031"
Aug 05 17:21:57.185217 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:21:57.185183    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-2b982"
Aug 05 17:22:22.187873 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:22.187841    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"
Aug 05 17:22:31.253968 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.253926    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"]
Aug 05 17:22:31.254458 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.254153    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5" podUID="494262a5-2678-44d1-a10c-da1a7df87e1c" containerName="manager" containerID="cri-o://8edf3192f46baa01ffd003c186e718ed0a021cad76d15ae9de1bd39edf01bb4a" gracePeriod=10
Aug 05 17:22:31.280841 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.280805    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-bvr5x"]
Aug 05 17:22:31.282779 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.282764    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f49dcd9ff-bvr5x"
Aug 05 17:22:31.291524 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.291498    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-bvr5x"]
Aug 05 17:22:31.299721 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.299693    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16a4a7de-e6a6-4260-a8a2-79a956677d62-cert\") pod \"kserve-controller-manager-6f49dcd9ff-bvr5x\" (UID: \"16a4a7de-e6a6-4260-a8a2-79a956677d62\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-bvr5x"
Aug 05 17:22:31.299863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.299756    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmml4\" (UniqueName: \"kubernetes.io/projected/16a4a7de-e6a6-4260-a8a2-79a956677d62-kube-api-access-cmml4\") pod \"kserve-controller-manager-6f49dcd9ff-bvr5x\" (UID: \"16a4a7de-e6a6-4260-a8a2-79a956677d62\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-bvr5x"
Aug 05 17:22:31.400916 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.400876    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16a4a7de-e6a6-4260-a8a2-79a956677d62-cert\") pod \"kserve-controller-manager-6f49dcd9ff-bvr5x\" (UID: \"16a4a7de-e6a6-4260-a8a2-79a956677d62\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-bvr5x"
Aug 05 17:22:31.401131 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.400944    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmml4\" (UniqueName: \"kubernetes.io/projected/16a4a7de-e6a6-4260-a8a2-79a956677d62-kube-api-access-cmml4\") pod \"kserve-controller-manager-6f49dcd9ff-bvr5x\" (UID: \"16a4a7de-e6a6-4260-a8a2-79a956677d62\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-bvr5x"
Aug 05 17:22:31.403332 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.403306    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16a4a7de-e6a6-4260-a8a2-79a956677d62-cert\") pod \"kserve-controller-manager-6f49dcd9ff-bvr5x\" (UID: \"16a4a7de-e6a6-4260-a8a2-79a956677d62\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-bvr5x"
Aug 05 17:22:31.410146 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.410121    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmml4\" (UniqueName: \"kubernetes.io/projected/16a4a7de-e6a6-4260-a8a2-79a956677d62-kube-api-access-cmml4\") pod \"kserve-controller-manager-6f49dcd9ff-bvr5x\" (UID: \"16a4a7de-e6a6-4260-a8a2-79a956677d62\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-bvr5x"
Aug 05 17:22:31.482924 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.482898    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"
Aug 05 17:22:31.501851 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.501823    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvmbk\" (UniqueName: \"kubernetes.io/projected/494262a5-2678-44d1-a10c-da1a7df87e1c-kube-api-access-dvmbk\") pod \"494262a5-2678-44d1-a10c-da1a7df87e1c\" (UID: \"494262a5-2678-44d1-a10c-da1a7df87e1c\") "
Aug 05 17:22:31.502009 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.501901    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/494262a5-2678-44d1-a10c-da1a7df87e1c-cert\") pod \"494262a5-2678-44d1-a10c-da1a7df87e1c\" (UID: \"494262a5-2678-44d1-a10c-da1a7df87e1c\") "
Aug 05 17:22:31.504039 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.503977    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/494262a5-2678-44d1-a10c-da1a7df87e1c-cert" (OuterVolumeSpecName: "cert") pod "494262a5-2678-44d1-a10c-da1a7df87e1c" (UID: "494262a5-2678-44d1-a10c-da1a7df87e1c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:22:31.504039 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.504020    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494262a5-2678-44d1-a10c-da1a7df87e1c-kube-api-access-dvmbk" (OuterVolumeSpecName: "kube-api-access-dvmbk") pod "494262a5-2678-44d1-a10c-da1a7df87e1c" (UID: "494262a5-2678-44d1-a10c-da1a7df87e1c"). InnerVolumeSpecName "kube-api-access-dvmbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:22:31.602508 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.602475    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvmbk\" (UniqueName: \"kubernetes.io/projected/494262a5-2678-44d1-a10c-da1a7df87e1c-kube-api-access-dvmbk\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:22:31.602508 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.602500    2574 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/494262a5-2678-44d1-a10c-da1a7df87e1c-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:22:31.630553 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.630524    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f49dcd9ff-bvr5x"
Aug 05 17:22:31.755012 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:31.754939    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-bvr5x"]
Aug 05 17:22:31.757946 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:22:31.757922    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a4a7de_e6a6_4260_a8a2_79a956677d62.slice/crio-99e021dc25565e2b3b326e6bdf46d891d2577ebb9f6fe55bceeb6350c0f07073 WatchSource:0}: Error finding container 99e021dc25565e2b3b326e6bdf46d891d2577ebb9f6fe55bceeb6350c0f07073: Status 404 returned error can't find the container with id 99e021dc25565e2b3b326e6bdf46d891d2577ebb9f6fe55bceeb6350c0f07073
Aug 05 17:22:32.310638 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:32.310538    2574 generic.go:356] "Generic (PLEG): container finished" podID="494262a5-2678-44d1-a10c-da1a7df87e1c" containerID="8edf3192f46baa01ffd003c186e718ed0a021cad76d15ae9de1bd39edf01bb4a" exitCode=0
Aug 05 17:22:32.310638 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:32.310592    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"
Aug 05 17:22:32.310638 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:32.310617    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5" event={"ID":"494262a5-2678-44d1-a10c-da1a7df87e1c","Type":"ContainerDied","Data":"8edf3192f46baa01ffd003c186e718ed0a021cad76d15ae9de1bd39edf01bb4a"}
Aug 05 17:22:32.311222 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:32.310671    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f49dcd9ff-8g9t5" event={"ID":"494262a5-2678-44d1-a10c-da1a7df87e1c","Type":"ContainerDied","Data":"50cb6acb17c44ff514cc3303a932703652cb44245eb7a0e4b66f3553e5e3b30e"}
Aug 05 17:22:32.311222 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:32.310702    2574 scope.go:117] "RemoveContainer" containerID="8edf3192f46baa01ffd003c186e718ed0a021cad76d15ae9de1bd39edf01bb4a"
Aug 05 17:22:32.312269 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:32.312243    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f49dcd9ff-bvr5x" event={"ID":"16a4a7de-e6a6-4260-a8a2-79a956677d62","Type":"ContainerStarted","Data":"0ad05a57d13a5ad8607b412dbe9c18e2123152652e429bc034827a83f85f39cd"}
Aug 05 17:22:32.312379 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:32.312293    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f49dcd9ff-bvr5x" event={"ID":"16a4a7de-e6a6-4260-a8a2-79a956677d62","Type":"ContainerStarted","Data":"99e021dc25565e2b3b326e6bdf46d891d2577ebb9f6fe55bceeb6350c0f07073"}
Aug 05 17:22:32.312379 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:32.312337    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6f49dcd9ff-bvr5x"
Aug 05 17:22:32.318334 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:32.318319    2574 scope.go:117] "RemoveContainer" containerID="8edf3192f46baa01ffd003c186e718ed0a021cad76d15ae9de1bd39edf01bb4a"
Aug 05 17:22:32.318598 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:22:32.318573    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8edf3192f46baa01ffd003c186e718ed0a021cad76d15ae9de1bd39edf01bb4a\": container with ID starting with 8edf3192f46baa01ffd003c186e718ed0a021cad76d15ae9de1bd39edf01bb4a not found: ID does not exist" containerID="8edf3192f46baa01ffd003c186e718ed0a021cad76d15ae9de1bd39edf01bb4a"
Aug 05 17:22:32.318646 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:32.318606    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8edf3192f46baa01ffd003c186e718ed0a021cad76d15ae9de1bd39edf01bb4a"} err="failed to get container status \"8edf3192f46baa01ffd003c186e718ed0a021cad76d15ae9de1bd39edf01bb4a\": rpc error: code = NotFound desc = could not find container \"8edf3192f46baa01ffd003c186e718ed0a021cad76d15ae9de1bd39edf01bb4a\": container with ID starting with 8edf3192f46baa01ffd003c186e718ed0a021cad76d15ae9de1bd39edf01bb4a not found: ID does not exist"
Aug 05 17:22:32.330648 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:32.330621    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"]
Aug 05 17:22:32.332540 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:32.332516    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-8g9t5"]
Aug 05 17:22:32.349284 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:32.349237    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6f49dcd9ff-bvr5x" podStartSLOduration=1.047735866 podStartE2EDuration="1.349222888s" totalImagesPullingTime="301.487022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:32.348387843 +0000 UTC m=+602.640242679" watchObservedRunningTime="2026-08-05 17:22:32.349222888 +0000 UTC m=+602.641077719"
Aug 05 17:22:34.193151 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:22:34.193117    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="494262a5-2678-44d1-a10c-da1a7df87e1c" path="/var/lib/kubelet/pods/494262a5-2678-44d1-a10c-da1a7df87e1c/volumes"
Aug 05 17:23:03.321629 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:03.321594    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6f49dcd9ff-bvr5x"
Aug 05 17:23:04.587953 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.587920    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-8588677cfb-z6fjd"]
Aug 05 17:23:04.588437 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.588273    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="494262a5-2678-44d1-a10c-da1a7df87e1c" containerName="manager"
Aug 05 17:23:04.588437 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.588289    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="494262a5-2678-44d1-a10c-da1a7df87e1c" containerName="manager"
Aug 05 17:23:04.588437 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.588355    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="494262a5-2678-44d1-a10c-da1a7df87e1c" containerName="manager"
Aug 05 17:23:04.590885 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.590866    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-z6fjd"
Aug 05 17:23:04.595144 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.595119    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Aug 05 17:23:04.595261 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.595241    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-pfsqk\""
Aug 05 17:23:04.595310 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.595244    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Aug 05 17:23:04.603254 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.603234    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-z6fjd"]
Aug 05 17:23:04.633110 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.633052    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc73af62-599e-42ec-aecd-51377ff21ae6-cert\") pod \"odh-model-controller-8588677cfb-z6fjd\" (UID: \"cc73af62-599e-42ec-aecd-51377ff21ae6\") " pod="kserve/odh-model-controller-8588677cfb-z6fjd"
Aug 05 17:23:04.633297 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.633189    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2ff7\" (UniqueName: \"kubernetes.io/projected/cc73af62-599e-42ec-aecd-51377ff21ae6-kube-api-access-j2ff7\") pod \"odh-model-controller-8588677cfb-z6fjd\" (UID: \"cc73af62-599e-42ec-aecd-51377ff21ae6\") " pod="kserve/odh-model-controller-8588677cfb-z6fjd"
Aug 05 17:23:04.633297 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.633223    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc73af62-599e-42ec-aecd-51377ff21ae6-metrics-certs\") pod \"odh-model-controller-8588677cfb-z6fjd\" (UID: \"cc73af62-599e-42ec-aecd-51377ff21ae6\") " pod="kserve/odh-model-controller-8588677cfb-z6fjd"
Aug 05 17:23:04.734573 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.734532    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc73af62-599e-42ec-aecd-51377ff21ae6-cert\") pod \"odh-model-controller-8588677cfb-z6fjd\" (UID: \"cc73af62-599e-42ec-aecd-51377ff21ae6\") " pod="kserve/odh-model-controller-8588677cfb-z6fjd"
Aug 05 17:23:04.734785 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.734589    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2ff7\" (UniqueName: \"kubernetes.io/projected/cc73af62-599e-42ec-aecd-51377ff21ae6-kube-api-access-j2ff7\") pod \"odh-model-controller-8588677cfb-z6fjd\" (UID: \"cc73af62-599e-42ec-aecd-51377ff21ae6\") " pod="kserve/odh-model-controller-8588677cfb-z6fjd"
Aug 05 17:23:04.734785 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.734607    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc73af62-599e-42ec-aecd-51377ff21ae6-metrics-certs\") pod \"odh-model-controller-8588677cfb-z6fjd\" (UID: \"cc73af62-599e-42ec-aecd-51377ff21ae6\") " pod="kserve/odh-model-controller-8588677cfb-z6fjd"
Aug 05 17:23:04.734785 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:23:04.734685    2574 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Aug 05 17:23:04.734785 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:23:04.734695    2574 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Aug 05 17:23:04.734785 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:23:04.734749    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc73af62-599e-42ec-aecd-51377ff21ae6-metrics-certs podName:cc73af62-599e-42ec-aecd-51377ff21ae6 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:05.234732023 +0000 UTC m=+635.526586832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc73af62-599e-42ec-aecd-51377ff21ae6-metrics-certs") pod "odh-model-controller-8588677cfb-z6fjd" (UID: "cc73af62-599e-42ec-aecd-51377ff21ae6") : secret "odh-model-controller-metrics-tls" not found
Aug 05 17:23:04.734785 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:23:04.734762    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc73af62-599e-42ec-aecd-51377ff21ae6-cert podName:cc73af62-599e-42ec-aecd-51377ff21ae6 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:05.234756329 +0000 UTC m=+635.526611138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/cc73af62-599e-42ec-aecd-51377ff21ae6-cert") pod "odh-model-controller-8588677cfb-z6fjd" (UID: "cc73af62-599e-42ec-aecd-51377ff21ae6") : secret "odh-model-controller-webhook-cert" not found
Aug 05 17:23:04.746931 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:04.746894    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2ff7\" (UniqueName: \"kubernetes.io/projected/cc73af62-599e-42ec-aecd-51377ff21ae6-kube-api-access-j2ff7\") pod \"odh-model-controller-8588677cfb-z6fjd\" (UID: \"cc73af62-599e-42ec-aecd-51377ff21ae6\") " pod="kserve/odh-model-controller-8588677cfb-z6fjd"
Aug 05 17:23:05.239139 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:05.239096    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc73af62-599e-42ec-aecd-51377ff21ae6-metrics-certs\") pod \"odh-model-controller-8588677cfb-z6fjd\" (UID: \"cc73af62-599e-42ec-aecd-51377ff21ae6\") " pod="kserve/odh-model-controller-8588677cfb-z6fjd"
Aug 05 17:23:05.239401 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:05.239376    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc73af62-599e-42ec-aecd-51377ff21ae6-cert\") pod \"odh-model-controller-8588677cfb-z6fjd\" (UID: \"cc73af62-599e-42ec-aecd-51377ff21ae6\") " pod="kserve/odh-model-controller-8588677cfb-z6fjd"
Aug 05 17:23:05.241746 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:05.241716    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc73af62-599e-42ec-aecd-51377ff21ae6-metrics-certs\") pod \"odh-model-controller-8588677cfb-z6fjd\" (UID: \"cc73af62-599e-42ec-aecd-51377ff21ae6\") " pod="kserve/odh-model-controller-8588677cfb-z6fjd"
Aug 05 17:23:05.241848 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:05.241716    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc73af62-599e-42ec-aecd-51377ff21ae6-cert\") pod \"odh-model-controller-8588677cfb-z6fjd\" (UID: \"cc73af62-599e-42ec-aecd-51377ff21ae6\") " pod="kserve/odh-model-controller-8588677cfb-z6fjd"
Aug 05 17:23:05.500589 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:05.500554    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-z6fjd"
Aug 05 17:23:05.630793 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:05.629557    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-z6fjd"]
Aug 05 17:23:05.635042 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:23:05.635014    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc73af62_599e_42ec_aecd_51377ff21ae6.slice/crio-75c74be6f02a7192dee8e3498e8157cf72e719c60e3417a57f519069059368fb WatchSource:0}: Error finding container 75c74be6f02a7192dee8e3498e8157cf72e719c60e3417a57f519069059368fb: Status 404 returned error can't find the container with id 75c74be6f02a7192dee8e3498e8157cf72e719c60e3417a57f519069059368fb
Aug 05 17:23:06.426803 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:06.426756    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-z6fjd" event={"ID":"cc73af62-599e-42ec-aecd-51377ff21ae6","Type":"ContainerStarted","Data":"75c74be6f02a7192dee8e3498e8157cf72e719c60e3417a57f519069059368fb"}
Aug 05 17:23:08.436526 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:08.436414    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-z6fjd" event={"ID":"cc73af62-599e-42ec-aecd-51377ff21ae6","Type":"ContainerStarted","Data":"e8ac636e596a6e6dc1981cbda1112ff18126a68f5bd8a00317c9b251b1f53a59"}
Aug 05 17:23:08.436948 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:08.436525    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-8588677cfb-z6fjd"
Aug 05 17:23:08.455892 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:08.455841    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-8588677cfb-z6fjd" podStartSLOduration=1.947750205 podStartE2EDuration="4.45582319s" totalImagesPullingTime="2.508072985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:23:08.454522152 +0000 UTC m=+638.746376983" watchObservedRunningTime="2026-08-05 17:23:08.45582319 +0000 UTC m=+638.747678021"
Aug 05 17:23:19.442612 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:19.442578    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-8588677cfb-z6fjd"
Aug 05 17:23:20.339557 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:20.339521    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-9snm8"]
Aug 05 17:23:20.342620 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:20.342603    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9snm8"
Aug 05 17:23:20.352182 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:20.352156    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9snm8"]
Aug 05 17:23:20.451718 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:20.451683    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgwsk\" (UniqueName: \"kubernetes.io/projected/a70bcb32-2274-4161-966f-51520adc4c91-kube-api-access-pgwsk\") pod \"s3-init-9snm8\" (UID: \"a70bcb32-2274-4161-966f-51520adc4c91\") " pod="kserve/s3-init-9snm8"
Aug 05 17:23:20.452158 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:20.451744    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a70bcb32-2274-4161-966f-51520adc4c91-model-cache\") pod \"s3-init-9snm8\" (UID: \"a70bcb32-2274-4161-966f-51520adc4c91\") " pod="kserve/s3-init-9snm8"
Aug 05 17:23:20.552671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:20.552631    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a70bcb32-2274-4161-966f-51520adc4c91-model-cache\") pod \"s3-init-9snm8\" (UID: \"a70bcb32-2274-4161-966f-51520adc4c91\") " pod="kserve/s3-init-9snm8"
Aug 05 17:23:20.552848 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:20.552683    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgwsk\" (UniqueName: \"kubernetes.io/projected/a70bcb32-2274-4161-966f-51520adc4c91-kube-api-access-pgwsk\") pod \"s3-init-9snm8\" (UID: \"a70bcb32-2274-4161-966f-51520adc4c91\") " pod="kserve/s3-init-9snm8"
Aug 05 17:23:20.553151 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:20.553131    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a70bcb32-2274-4161-966f-51520adc4c91-model-cache\") pod \"s3-init-9snm8\" (UID: \"a70bcb32-2274-4161-966f-51520adc4c91\") " pod="kserve/s3-init-9snm8"
Aug 05 17:23:20.562210 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:20.562185    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgwsk\" (UniqueName: \"kubernetes.io/projected/a70bcb32-2274-4161-966f-51520adc4c91-kube-api-access-pgwsk\") pod \"s3-init-9snm8\" (UID: \"a70bcb32-2274-4161-966f-51520adc4c91\") " pod="kserve/s3-init-9snm8"
Aug 05 17:23:20.651753 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:20.651650    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9snm8"
Aug 05 17:23:20.783021 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:20.782836    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9snm8"]
Aug 05 17:23:20.785774 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:23:20.785747    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda70bcb32_2274_4161_966f_51520adc4c91.slice/crio-9eaec742b2c08d806ed7a2a9903932034c56d008aec915dfc6da0d9d7e08c143 WatchSource:0}: Error finding container 9eaec742b2c08d806ed7a2a9903932034c56d008aec915dfc6da0d9d7e08c143: Status 404 returned error can't find the container with id 9eaec742b2c08d806ed7a2a9903932034c56d008aec915dfc6da0d9d7e08c143
Aug 05 17:23:21.488306 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:21.488264    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9snm8" event={"ID":"a70bcb32-2274-4161-966f-51520adc4c91","Type":"ContainerStarted","Data":"9eaec742b2c08d806ed7a2a9903932034c56d008aec915dfc6da0d9d7e08c143"}
Aug 05 17:23:24.501692 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:24.501658    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9snm8" event={"ID":"a70bcb32-2274-4161-966f-51520adc4c91","Type":"ContainerStarted","Data":"90a18914ee76726f2e8cbf628382584229d7974b0c8545133e051993556c5590"}
Aug 05 17:23:26.509956 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:26.509919    2574 generic.go:356] "Generic (PLEG): container finished" podID="a70bcb32-2274-4161-966f-51520adc4c91" containerID="90a18914ee76726f2e8cbf628382584229d7974b0c8545133e051993556c5590" exitCode=0
Aug 05 17:23:26.510351 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:26.509992    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9snm8" event={"ID":"a70bcb32-2274-4161-966f-51520adc4c91","Type":"ContainerDied","Data":"90a18914ee76726f2e8cbf628382584229d7974b0c8545133e051993556c5590"}
Aug 05 17:23:31.532692 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:31.532652    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9snm8" event={"ID":"a70bcb32-2274-4161-966f-51520adc4c91","Type":"ContainerStarted","Data":"8773797eedd3db04202455d546bcdcd6d4b76b10ee5395191aa5663f727554af"}
Aug 05 17:23:31.552287 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:23:31.552233    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-9snm8" podStartSLOduration=2.339355111 podStartE2EDuration="11.55221579s" totalImagesPullingTime="7.685110375s" totalInitContainerRuntime="1.527750304s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:23:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:23:31.550721428 +0000 UTC m=+661.842576263" watchObservedRunningTime="2026-08-05 17:23:31.55221579 +0000 UTC m=+661.844070649"
Aug 05 17:24:36.752886 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:36.752851    2574 generic.go:356] "Generic (PLEG): container finished" podID="a70bcb32-2274-4161-966f-51520adc4c91" containerID="8773797eedd3db04202455d546bcdcd6d4b76b10ee5395191aa5663f727554af" exitCode=0
Aug 05 17:24:36.753340 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:36.752928    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9snm8" event={"ID":"a70bcb32-2274-4161-966f-51520adc4c91","Type":"ContainerDied","Data":"8773797eedd3db04202455d546bcdcd6d4b76b10ee5395191aa5663f727554af"}
Aug 05 17:24:37.877873 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:37.877848    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9snm8"
Aug 05 17:24:37.908590 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:37.908559    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a70bcb32-2274-4161-966f-51520adc4c91-model-cache\") pod \"a70bcb32-2274-4161-966f-51520adc4c91\" (UID: \"a70bcb32-2274-4161-966f-51520adc4c91\") "
Aug 05 17:24:37.908755 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:37.908699    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgwsk\" (UniqueName: \"kubernetes.io/projected/a70bcb32-2274-4161-966f-51520adc4c91-kube-api-access-pgwsk\") pod \"a70bcb32-2274-4161-966f-51520adc4c91\" (UID: \"a70bcb32-2274-4161-966f-51520adc4c91\") "
Aug 05 17:24:37.910726 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:37.910690    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70bcb32-2274-4161-966f-51520adc4c91-kube-api-access-pgwsk" (OuterVolumeSpecName: "kube-api-access-pgwsk") pod "a70bcb32-2274-4161-966f-51520adc4c91" (UID: "a70bcb32-2274-4161-966f-51520adc4c91"). InnerVolumeSpecName "kube-api-access-pgwsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:24:37.927680 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:37.927628    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70bcb32-2274-4161-966f-51520adc4c91-model-cache" (OuterVolumeSpecName: "model-cache") pod "a70bcb32-2274-4161-966f-51520adc4c91" (UID: "a70bcb32-2274-4161-966f-51520adc4c91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:24:38.010123 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:38.010018    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgwsk\" (UniqueName: \"kubernetes.io/projected/a70bcb32-2274-4161-966f-51520adc4c91-kube-api-access-pgwsk\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:24:38.010123 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:38.010048    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a70bcb32-2274-4161-966f-51520adc4c91-model-cache\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:24:38.760931 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:38.760904    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9snm8"
Aug 05 17:24:38.761110 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:38.760903    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9snm8" event={"ID":"a70bcb32-2274-4161-966f-51520adc4c91","Type":"ContainerDied","Data":"9eaec742b2c08d806ed7a2a9903932034c56d008aec915dfc6da0d9d7e08c143"}
Aug 05 17:24:38.761110 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:38.761021    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eaec742b2c08d806ed7a2a9903932034c56d008aec915dfc6da0d9d7e08c143"
Aug 05 17:24:52.971182 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:52.971145    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"]
Aug 05 17:24:52.971656 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:52.971389    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a70bcb32-2274-4161-966f-51520adc4c91" containerName="s3-init"
Aug 05 17:24:52.971656 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:52.971399    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a70bcb32-2274-4161-966f-51520adc4c91" containerName="s3-init"
Aug 05 17:24:52.971656 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:52.971409    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a70bcb32-2274-4161-966f-51520adc4c91" containerName="download-hf-model"
Aug 05 17:24:52.971656 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:52.971415    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a70bcb32-2274-4161-966f-51520adc4c91" containerName="download-hf-model"
Aug 05 17:24:52.971656 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:52.971460    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a70bcb32-2274-4161-966f-51520adc4c91" containerName="s3-init"
Aug 05 17:24:52.974781 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:52.974764    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:52.977785 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:52.977748    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 17:24:52.977785 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:52.977755    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-fdd1f-predictor-serving-cert\""
Aug 05 17:24:52.977973 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:52.977788    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-fdd1f-kube-rbac-proxy-sar-config\""
Aug 05 17:24:52.977973 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:52.977806    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 05 17:24:52.977973 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:52.977788    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-h9rpc\""
Aug 05 17:24:52.977973 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:52.977888    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 05 17:24:52.978262 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:52.978249    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 05 17:24:52.997217 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:52.997182    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"]
Aug 05 17:24:53.018235 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.018202    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.018415 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.018243    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64d21b6a-55de-40b5-b65f-0acca1fc29b9-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.018415 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.018268    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64d21b6a-55de-40b5-b65f-0acca1fc29b9-proxy-tls\") pod \"isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.018415 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.018346    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zjqb\" (UniqueName: \"kubernetes.io/projected/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kube-api-access-5zjqb\") pod \"isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.018415 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.018399    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.119323 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.119290    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64d21b6a-55de-40b5-b65f-0acca1fc29b9-proxy-tls\") pod \"isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.119323 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.119331    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zjqb\" (UniqueName: \"kubernetes.io/projected/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kube-api-access-5zjqb\") pod \"isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.119554 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.119353    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.119554 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.119410    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.119554 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.119436    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64d21b6a-55de-40b5-b65f-0acca1fc29b9-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.119886 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.119867    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.120086 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.120059    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.120218 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.120197    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64d21b6a-55de-40b5-b65f-0acca1fc29b9-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.121853 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.121837    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64d21b6a-55de-40b5-b65f-0acca1fc29b9-proxy-tls\") pod \"isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.133974 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.133946    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zjqb\" (UniqueName: \"kubernetes.io/projected/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kube-api-access-5zjqb\") pod \"isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.183755 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.183718    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"]
Aug 05 17:24:53.187055 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.187033    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.189736 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.189714    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Aug 05 17:24:53.189858 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.189799    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Aug 05 17:24:53.209689 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.209659    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"]
Aug 05 17:24:53.253016 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.252980    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"]
Aug 05 17:24:53.256267 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.256244    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.259868 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.259848    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Aug 05 17:24:53.272997 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.272971    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"]
Aug 05 17:24:53.284360 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.284332    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:24:53.320530 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.320498    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55c3126-506b-4819-b0d1-a90709270977-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.320530 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.320537    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bba18d24-a734-4f0f-80ce-e78439c474c7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.320790 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.320569    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bba18d24-a734-4f0f-80ce-e78439c474c7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.320790 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.320631    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c55c3126-506b-4819-b0d1-a90709270977-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.320790 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.320701    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bba18d24-a734-4f0f-80ce-e78439c474c7-proxy-tls\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.320790 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.320743    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjrv\" (UniqueName: \"kubernetes.io/projected/c55c3126-506b-4819-b0d1-a90709270977-kube-api-access-2sjrv\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.320790 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.320772    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c55c3126-506b-4819-b0d1-a90709270977-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.321024 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.320809    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c55c3126-506b-4819-b0d1-a90709270977-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.321024 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.320849    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bba18d24-a734-4f0f-80ce-e78439c474c7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.321024 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.320873    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfg67\" (UniqueName: \"kubernetes.io/projected/bba18d24-a734-4f0f-80ce-e78439c474c7-kube-api-access-lfg67\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.421472 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.421438    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55c3126-506b-4819-b0d1-a90709270977-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.421472 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.421478    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bba18d24-a734-4f0f-80ce-e78439c474c7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.421739 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.421504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bba18d24-a734-4f0f-80ce-e78439c474c7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.421739 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.421536    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c55c3126-506b-4819-b0d1-a90709270977-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.421739 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.421569    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bba18d24-a734-4f0f-80ce-e78439c474c7-proxy-tls\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.421739 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.421592    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjrv\" (UniqueName: \"kubernetes.io/projected/c55c3126-506b-4819-b0d1-a90709270977-kube-api-access-2sjrv\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.421739 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.421618    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c55c3126-506b-4819-b0d1-a90709270977-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.421739 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.421651    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c55c3126-506b-4819-b0d1-a90709270977-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.421739 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.421686    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bba18d24-a734-4f0f-80ce-e78439c474c7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.421739 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.421711    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfg67\" (UniqueName: \"kubernetes.io/projected/bba18d24-a734-4f0f-80ce-e78439c474c7-kube-api-access-lfg67\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.421739 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:24:53.421715    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Aug 05 17:24:53.422196 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:24:53.421798    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bba18d24-a734-4f0f-80ce-e78439c474c7-proxy-tls podName:bba18d24-a734-4f0f-80ce-e78439c474c7 nodeName:}" failed. No retries permitted until 2026-08-05 17:24:53.921775102 +0000 UTC m=+744.213629924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bba18d24-a734-4f0f-80ce-e78439c474c7-proxy-tls") pod "isvc-canary-rollback-predictor-55cddd885b-kmz8m" (UID: "bba18d24-a734-4f0f-80ce-e78439c474c7") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Aug 05 17:24:53.422196 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.421916    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55c3126-506b-4819-b0d1-a90709270977-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.422196 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.421916    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bba18d24-a734-4f0f-80ce-e78439c474c7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.422196 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:24:53.422022    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Aug 05 17:24:53.422196 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:24:53.422094    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c55c3126-506b-4819-b0d1-a90709270977-proxy-tls podName:c55c3126-506b-4819-b0d1-a90709270977 nodeName:}" failed. No retries permitted until 2026-08-05 17:24:53.922058672 +0000 UTC m=+744.213913487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c55c3126-506b-4819-b0d1-a90709270977-proxy-tls") pod "isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" (UID: "c55c3126-506b-4819-b0d1-a90709270977") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Aug 05 17:24:53.422464 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.422370    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c55c3126-506b-4819-b0d1-a90709270977-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.422600 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.422577    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bba18d24-a734-4f0f-80ce-e78439c474c7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.422661 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.422621    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c55c3126-506b-4819-b0d1-a90709270977-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.422711 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.422657    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bba18d24-a734-4f0f-80ce-e78439c474c7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.427941 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.427919    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"]
Aug 05 17:24:53.431342 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:24:53.431308    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d21b6a_55de_40b5_b65f_0acca1fc29b9.slice/crio-457678e33d3487cd3738d282588a5e1e5f62286f8345e06672a5bf4603db4027 WatchSource:0}: Error finding container 457678e33d3487cd3738d282588a5e1e5f62286f8345e06672a5bf4603db4027: Status 404 returned error can't find the container with id 457678e33d3487cd3738d282588a5e1e5f62286f8345e06672a5bf4603db4027
Aug 05 17:24:53.433904 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.433876    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfg67\" (UniqueName: \"kubernetes.io/projected/bba18d24-a734-4f0f-80ce-e78439c474c7-kube-api-access-lfg67\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.434017 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.433918    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjrv\" (UniqueName: \"kubernetes.io/projected/c55c3126-506b-4819-b0d1-a90709270977-kube-api-access-2sjrv\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.815519 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.815475    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" event={"ID":"64d21b6a-55de-40b5-b65f-0acca1fc29b9","Type":"ContainerStarted","Data":"2f322729ce85b12fa6dedd11a0ca1a97ed23c3b4685897d3dbcf664146c44cca"}
Aug 05 17:24:53.815721 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.815560    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" event={"ID":"64d21b6a-55de-40b5-b65f-0acca1fc29b9","Type":"ContainerStarted","Data":"457678e33d3487cd3738d282588a5e1e5f62286f8345e06672a5bf4603db4027"}
Aug 05 17:24:53.924702 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.924662    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bba18d24-a734-4f0f-80ce-e78439c474c7-proxy-tls\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:53.924890 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.924726    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c55c3126-506b-4819-b0d1-a90709270977-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.927726 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.927703    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c55c3126-506b-4819-b0d1-a90709270977-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:53.927864 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:53.927840    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bba18d24-a734-4f0f-80ce-e78439c474c7-proxy-tls\") pod \"isvc-canary-rollback-predictor-55cddd885b-kmz8m\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:54.097729 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:54.097649    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:24:54.166957 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:54.166923    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:24:54.244700 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:54.244668    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"]
Aug 05 17:24:54.248488 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:24:54.248449    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba18d24_a734_4f0f_80ce_e78439c474c7.slice/crio-7a1875c75916ea1ec7465684a858d3441b41606a058e632063ea640136d6ebfa WatchSource:0}: Error finding container 7a1875c75916ea1ec7465684a858d3441b41606a058e632063ea640136d6ebfa: Status 404 returned error can't find the container with id 7a1875c75916ea1ec7465684a858d3441b41606a058e632063ea640136d6ebfa
Aug 05 17:24:54.323304 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:54.323275    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"]
Aug 05 17:24:54.326634 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:24:54.326602    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55c3126_506b_4819_b0d1_a90709270977.slice/crio-8a77e9a7662ce48d9531b30b246dde6521ac8c57e074c42301349402d9e95c37 WatchSource:0}: Error finding container 8a77e9a7662ce48d9531b30b246dde6521ac8c57e074c42301349402d9e95c37: Status 404 returned error can't find the container with id 8a77e9a7662ce48d9531b30b246dde6521ac8c57e074c42301349402d9e95c37
Aug 05 17:24:54.821870 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:54.821810    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" event={"ID":"c55c3126-506b-4819-b0d1-a90709270977","Type":"ContainerStarted","Data":"44482309e1a8b989839ada65c5b8249a7bb718e8856f1b1265112bf1ce5d99a8"}
Aug 05 17:24:54.822099 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:54.821913    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" event={"ID":"c55c3126-506b-4819-b0d1-a90709270977","Type":"ContainerStarted","Data":"8a77e9a7662ce48d9531b30b246dde6521ac8c57e074c42301349402d9e95c37"}
Aug 05 17:24:54.823521 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:54.823496    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" event={"ID":"bba18d24-a734-4f0f-80ce-e78439c474c7","Type":"ContainerStarted","Data":"97f278efb2e756f54bf48830778722622be5703ae800c733e996c70961e414c0"}
Aug 05 17:24:54.823639 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:54.823562    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" event={"ID":"bba18d24-a734-4f0f-80ce-e78439c474c7","Type":"ContainerStarted","Data":"7a1875c75916ea1ec7465684a858d3441b41606a058e632063ea640136d6ebfa"}
Aug 05 17:24:57.834791 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:57.834759    2574 generic.go:356] "Generic (PLEG): container finished" podID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerID="2f322729ce85b12fa6dedd11a0ca1a97ed23c3b4685897d3dbcf664146c44cca" exitCode=0
Aug 05 17:24:57.835192 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:57.834825    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" event={"ID":"64d21b6a-55de-40b5-b65f-0acca1fc29b9","Type":"ContainerDied","Data":"2f322729ce85b12fa6dedd11a0ca1a97ed23c3b4685897d3dbcf664146c44cca"}
Aug 05 17:24:58.842361 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:58.842315    2574 generic.go:356] "Generic (PLEG): container finished" podID="c55c3126-506b-4819-b0d1-a90709270977" containerID="44482309e1a8b989839ada65c5b8249a7bb718e8856f1b1265112bf1ce5d99a8" exitCode=0
Aug 05 17:24:58.842847 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:58.842422    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" event={"ID":"c55c3126-506b-4819-b0d1-a90709270977","Type":"ContainerDied","Data":"44482309e1a8b989839ada65c5b8249a7bb718e8856f1b1265112bf1ce5d99a8"}
Aug 05 17:24:58.847005 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:58.846177    2574 generic.go:356] "Generic (PLEG): container finished" podID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerID="97f278efb2e756f54bf48830778722622be5703ae800c733e996c70961e414c0" exitCode=0
Aug 05 17:24:58.847005 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:24:58.846248    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" event={"ID":"bba18d24-a734-4f0f-80ce-e78439c474c7","Type":"ContainerDied","Data":"97f278efb2e756f54bf48830778722622be5703ae800c733e996c70961e414c0"}
Aug 05 17:25:18.018186 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:18.018064    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" event={"ID":"c55c3126-506b-4819-b0d1-a90709270977","Type":"ContainerStarted","Data":"6fe541318ad9653eb73494dafb1f32f7549fe72eb2b3919ca518d8cb3da62668"}
Aug 05 17:25:18.022084 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:18.022032    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" event={"ID":"bba18d24-a734-4f0f-80ce-e78439c474c7","Type":"ContainerStarted","Data":"2f51df0780d90bc4663cbb76754dc9bcd3e6f7727e3bac6e87d87aff519fa202"}
Aug 05 17:25:18.027583 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:18.027527    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" event={"ID":"64d21b6a-55de-40b5-b65f-0acca1fc29b9","Type":"ContainerStarted","Data":"56662a61bc587fce9ccff2e79f9d0f768b0c6f8a4a8df2136027c0fd20b12451"}
Aug 05 17:25:21.044373 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:21.044336    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" event={"ID":"64d21b6a-55de-40b5-b65f-0acca1fc29b9","Type":"ContainerStarted","Data":"b2a5da8da8234ad63c3592fcd4be805b70a90f2ae2478de692326b6daac8b50d"}
Aug 05 17:25:21.046400 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:21.046368    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" event={"ID":"c55c3126-506b-4819-b0d1-a90709270977","Type":"ContainerStarted","Data":"ca289200789da3447a55f88cbcfaddcfa6bdcb799aec2f4fb5e6752e2598475b"}
Aug 05 17:25:21.046534 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:21.046501    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:25:21.048442 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:21.048419    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" event={"ID":"bba18d24-a734-4f0f-80ce-e78439c474c7","Type":"ContainerStarted","Data":"c51584f79ba2898f8c0fdb7e880ff148a80bbb1eb7b18380f9173bf0a266edeb"}
Aug 05 17:25:21.048569 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:21.048554    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:25:21.072257 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:21.070713    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" podStartSLOduration=2.916391074 podStartE2EDuration="28.070682128s" totalImagesPullingTime="21.38464915s" totalInitContainerRuntime="3.769641904s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:24:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:25:21.068548491 +0000 UTC m=+771.360403316" watchObservedRunningTime="2026-08-05 17:25:21.070682128 +0000 UTC m=+771.362536959"
Aug 05 17:25:21.093888 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:21.093822    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" podStartSLOduration=2.819421243 podStartE2EDuration="28.093804771s" totalImagesPullingTime="21.405778688s" totalInitContainerRuntime="3.86860484s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:24:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:25:21.092359596 +0000 UTC m=+771.384214428" watchObservedRunningTime="2026-08-05 17:25:21.093804771 +0000 UTC m=+771.385659602"
Aug 05 17:25:22.052681 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:22.052634    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:25:22.053155 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:22.053030    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:25:22.053782 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:22.053752    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Aug 05 17:25:22.054155 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:22.054128    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Aug 05 17:25:23.057096 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:23.057003    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Aug 05 17:25:23.057618 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:23.057121    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Aug 05 17:25:24.063727 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:24.063695    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" event={"ID":"64d21b6a-55de-40b5-b65f-0acca1fc29b9","Type":"ContainerStarted","Data":"258004ad4e5cc26ceae581c281f9f188af60bd60625eb4806b884360192c6477"}
Aug 05 17:25:24.064123 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:24.063905    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:25:24.064123 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:24.064029    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:25:24.065335 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:24.065309    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Aug 05 17:25:24.087846 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:24.087747    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podStartSLOduration=2.608433119 podStartE2EDuration="32.087713049s" totalImagesPullingTime="25.827197193s" totalInitContainerRuntime="3.652082737s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:24:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:25:24.086564903 +0000 UTC m=+774.378419735" watchObservedRunningTime="2026-08-05 17:25:24.087713049 +0000 UTC m=+774.379567883"
Aug 05 17:25:25.068142 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:25.068106    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:25:25.068541 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:25.068249    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Aug 05 17:25:25.069183 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:25.069152    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:25:25.071995 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:25.071974    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:25:26.072196 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:26.072158    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Aug 05 17:25:26.072599 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:26.072443    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:25:27.075960 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:27.075920    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Aug 05 17:25:27.076378 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:27.076350    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:25:28.062287 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:28.062254    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:25:28.062625 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:28.062607    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:25:28.062879 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:28.062849    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Aug 05 17:25:28.063390 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:28.063366    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Aug 05 17:25:37.076919 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:37.076867    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Aug 05 17:25:37.077419 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:37.077395    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:25:38.066421 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:38.066382    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Aug 05 17:25:38.066621 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:38.066375    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Aug 05 17:25:47.076556 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:47.076498    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Aug 05 17:25:47.076971 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:47.076947    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:25:48.063333 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:48.063289    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Aug 05 17:25:48.063543 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:48.063289    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Aug 05 17:25:57.076184 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:57.076043    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Aug 05 17:25:57.076709 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:57.076573    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:25:58.063746 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:58.063708    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Aug 05 17:25:58.063919 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:25:58.063715    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Aug 05 17:26:07.076070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:07.076002    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Aug 05 17:26:07.076634 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:07.076556    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:26:08.063633 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:08.063596    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Aug 05 17:26:08.063833 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:08.063596    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Aug 05 17:26:17.075994 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:17.075944    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Aug 05 17:26:17.076484 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:17.076456    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:26:18.062965 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:18.062926    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Aug 05 17:26:18.063319 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:18.063293    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Aug 05 17:26:27.076894 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:27.076845    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Aug 05 17:26:27.077375 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:27.077347    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:26:28.062770 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:28.062731    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Aug 05 17:26:28.063308 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:28.063281    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Aug 05 17:26:37.076843 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:37.076812    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:26:37.077269 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:37.077010    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:26:38.063774 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:38.063741    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:26:38.063954 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:38.063863    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:26:42.779014 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:42.778975    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"]
Aug 05 17:26:42.779518 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:42.779244    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kserve-container" containerID="cri-o://6fe541318ad9653eb73494dafb1f32f7549fe72eb2b3919ca518d8cb3da62668" gracePeriod=30
Aug 05 17:26:42.779518 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:42.779281    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kube-rbac-proxy" containerID="cri-o://ca289200789da3447a55f88cbcfaddcfa6bdcb799aec2f4fb5e6752e2598475b" gracePeriod=30
Aug 05 17:26:43.057481 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:43.057377    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Aug 05 17:26:43.365235 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:43.365144    2574 generic.go:356] "Generic (PLEG): container finished" podID="c55c3126-506b-4819-b0d1-a90709270977" containerID="ca289200789da3447a55f88cbcfaddcfa6bdcb799aec2f4fb5e6752e2598475b" exitCode=2
Aug 05 17:26:43.365235 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:43.365191    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" event={"ID":"c55c3126-506b-4819-b0d1-a90709270977","Type":"ContainerDied","Data":"ca289200789da3447a55f88cbcfaddcfa6bdcb799aec2f4fb5e6752e2598475b"}
Aug 05 17:26:47.722794 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.722768    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:26:47.797068 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.796986    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c55c3126-506b-4819-b0d1-a90709270977-proxy-tls\") pod \"c55c3126-506b-4819-b0d1-a90709270977\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") "
Aug 05 17:26:47.797068 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.797051    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55c3126-506b-4819-b0d1-a90709270977-kserve-provision-location\") pod \"c55c3126-506b-4819-b0d1-a90709270977\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") "
Aug 05 17:26:47.797327 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.797091    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c55c3126-506b-4819-b0d1-a90709270977-cabundle-cert\") pod \"c55c3126-506b-4819-b0d1-a90709270977\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") "
Aug 05 17:26:47.797327 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.797130    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sjrv\" (UniqueName: \"kubernetes.io/projected/c55c3126-506b-4819-b0d1-a90709270977-kube-api-access-2sjrv\") pod \"c55c3126-506b-4819-b0d1-a90709270977\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") "
Aug 05 17:26:47.797327 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.797146    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c55c3126-506b-4819-b0d1-a90709270977-kube-rbac-proxy-sar-config\") pod \"c55c3126-506b-4819-b0d1-a90709270977\" (UID: \"c55c3126-506b-4819-b0d1-a90709270977\") "
Aug 05 17:26:47.797488 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.797397    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55c3126-506b-4819-b0d1-a90709270977-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c55c3126-506b-4819-b0d1-a90709270977" (UID: "c55c3126-506b-4819-b0d1-a90709270977"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:26:47.797540 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.797493    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55c3126-506b-4819-b0d1-a90709270977-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c55c3126-506b-4819-b0d1-a90709270977" (UID: "c55c3126-506b-4819-b0d1-a90709270977"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:26:47.797576 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.797545    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55c3126-506b-4819-b0d1-a90709270977-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c55c3126-506b-4819-b0d1-a90709270977" (UID: "c55c3126-506b-4819-b0d1-a90709270977"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:26:47.799314 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.799286    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55c3126-506b-4819-b0d1-a90709270977-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c55c3126-506b-4819-b0d1-a90709270977" (UID: "c55c3126-506b-4819-b0d1-a90709270977"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:26:47.799424 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.799376    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55c3126-506b-4819-b0d1-a90709270977-kube-api-access-2sjrv" (OuterVolumeSpecName: "kube-api-access-2sjrv") pod "c55c3126-506b-4819-b0d1-a90709270977" (UID: "c55c3126-506b-4819-b0d1-a90709270977"). InnerVolumeSpecName "kube-api-access-2sjrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:26:47.860198 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.859775    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"]
Aug 05 17:26:47.861406 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.860376    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container" containerID="cri-o://56662a61bc587fce9ccff2e79f9d0f768b0c6f8a4a8df2136027c0fd20b12451" gracePeriod=30
Aug 05 17:26:47.861406 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.860700    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent" containerID="cri-o://258004ad4e5cc26ceae581c281f9f188af60bd60625eb4806b884360192c6477" gracePeriod=30
Aug 05 17:26:47.861406 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.860780    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kube-rbac-proxy" containerID="cri-o://b2a5da8da8234ad63c3592fcd4be805b70a90f2ae2478de692326b6daac8b50d" gracePeriod=30
Aug 05 17:26:47.898310 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.898273    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55c3126-506b-4819-b0d1-a90709270977-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:26:47.898310 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.898301    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c55c3126-506b-4819-b0d1-a90709270977-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:26:47.898310 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.898311    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sjrv\" (UniqueName: \"kubernetes.io/projected/c55c3126-506b-4819-b0d1-a90709270977-kube-api-access-2sjrv\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:26:47.898310 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.898319    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c55c3126-506b-4819-b0d1-a90709270977-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:26:47.898565 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.898330    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c55c3126-506b-4819-b0d1-a90709270977-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:26:47.962609 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.962575    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"]
Aug 05 17:26:47.962856 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.962844    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kube-rbac-proxy"
Aug 05 17:26:47.962899 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.962857    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kube-rbac-proxy"
Aug 05 17:26:47.962899 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.962875    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="storage-initializer"
Aug 05 17:26:47.962899 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.962881    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="storage-initializer"
Aug 05 17:26:47.962899 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.962888    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kserve-container"
Aug 05 17:26:47.962899 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.962894    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kserve-container"
Aug 05 17:26:47.963044 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.962941    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kserve-container"
Aug 05 17:26:47.963044 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.962949    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c55c3126-506b-4819-b0d1-a90709270977" containerName="kube-rbac-proxy"
Aug 05 17:26:47.965106 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.965066    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:47.967996 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.967969    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-849e6-predictor-serving-cert\""
Aug 05 17:26:47.968129 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.967975    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-849e6-kube-rbac-proxy-sar-config\""
Aug 05 17:26:47.980951 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:47.980929    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"]
Aug 05 17:26:48.065184 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.065113    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"]
Aug 05 17:26:48.067429 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.067406    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.069988 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.069968    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-849e6-kube-rbac-proxy-sar-config\""
Aug 05 17:26:48.069988 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.069979    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-849e6-predictor-serving-cert\""
Aug 05 17:26:48.080399 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.080376    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"]
Aug 05 17:26:48.099724 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.099690    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e152081d-0127-44e1-907d-751da8caab98-cabundle-cert\") pod \"isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.099876 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.099730    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kghvj\" (UniqueName: \"kubernetes.io/projected/e152081d-0127-44e1-907d-751da8caab98-kube-api-access-kghvj\") pod \"isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.099876 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.099754    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e152081d-0127-44e1-907d-751da8caab98-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.099876 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.099790    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e152081d-0127-44e1-907d-751da8caab98-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.099975 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.099871    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e152081d-0127-44e1-907d-751da8caab98-proxy-tls\") pod \"isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.200805 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.200772    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e152081d-0127-44e1-907d-751da8caab98-cabundle-cert\") pod \"isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.200962 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.200817    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kghvj\" (UniqueName: \"kubernetes.io/projected/e152081d-0127-44e1-907d-751da8caab98-kube-api-access-kghvj\") pod \"isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.200962 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.200845    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e152081d-0127-44e1-907d-751da8caab98-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.200962 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.200877    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx6vl\" (UniqueName: \"kubernetes.io/projected/83a40616-0eec-4d59-8776-95c7c986bb92-kube-api-access-gx6vl\") pod \"isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.200962 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.200903    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83a40616-0eec-4d59-8776-95c7c986bb92-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.200962 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.200952    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83a40616-0eec-4d59-8776-95c7c986bb92-proxy-tls\") pod \"isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.201376 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.200996    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e152081d-0127-44e1-907d-751da8caab98-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.201376 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.201114    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e152081d-0127-44e1-907d-751da8caab98-proxy-tls\") pod \"isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.201376 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.201147    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83a40616-0eec-4d59-8776-95c7c986bb92-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.201376 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.201186    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83a40616-0eec-4d59-8776-95c7c986bb92-cabundle-cert\") pod \"isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.201670 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.201521    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e152081d-0127-44e1-907d-751da8caab98-cabundle-cert\") pod \"isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.201712 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.201682    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e152081d-0127-44e1-907d-751da8caab98-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.201797 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.201781    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e152081d-0127-44e1-907d-751da8caab98-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.203542 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.203523    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e152081d-0127-44e1-907d-751da8caab98-proxy-tls\") pod \"isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.210876 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.210856    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kghvj\" (UniqueName: \"kubernetes.io/projected/e152081d-0127-44e1-907d-751da8caab98-kube-api-access-kghvj\") pod \"isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.275912 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.275872    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:48.302433 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.302395    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83a40616-0eec-4d59-8776-95c7c986bb92-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.302619 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.302443    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83a40616-0eec-4d59-8776-95c7c986bb92-cabundle-cert\") pod \"isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.302619 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.302497    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx6vl\" (UniqueName: \"kubernetes.io/projected/83a40616-0eec-4d59-8776-95c7c986bb92-kube-api-access-gx6vl\") pod \"isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.302619 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.302525    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83a40616-0eec-4d59-8776-95c7c986bb92-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.302619 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.302553    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83a40616-0eec-4d59-8776-95c7c986bb92-proxy-tls\") pod \"isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.303064 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.303036    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83a40616-0eec-4d59-8776-95c7c986bb92-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.303216 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.303174    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83a40616-0eec-4d59-8776-95c7c986bb92-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.303280 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.303262    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83a40616-0eec-4d59-8776-95c7c986bb92-cabundle-cert\") pod \"isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.305032 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.305011    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83a40616-0eec-4d59-8776-95c7c986bb92-proxy-tls\") pod \"isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.312188 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.312165    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx6vl\" (UniqueName: \"kubernetes.io/projected/83a40616-0eec-4d59-8776-95c7c986bb92-kube-api-access-gx6vl\") pod \"isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.377676 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.377640    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:26:48.387961 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.387930    2574 generic.go:356] "Generic (PLEG): container finished" podID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerID="b2a5da8da8234ad63c3592fcd4be805b70a90f2ae2478de692326b6daac8b50d" exitCode=2
Aug 05 17:26:48.388159 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.388013    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" event={"ID":"64d21b6a-55de-40b5-b65f-0acca1fc29b9","Type":"ContainerDied","Data":"b2a5da8da8234ad63c3592fcd4be805b70a90f2ae2478de692326b6daac8b50d"}
Aug 05 17:26:48.390097 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.390057    2574 generic.go:356] "Generic (PLEG): container finished" podID="c55c3126-506b-4819-b0d1-a90709270977" containerID="6fe541318ad9653eb73494dafb1f32f7549fe72eb2b3919ca518d8cb3da62668" exitCode=0
Aug 05 17:26:48.390209 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.390106    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" event={"ID":"c55c3126-506b-4819-b0d1-a90709270977","Type":"ContainerDied","Data":"6fe541318ad9653eb73494dafb1f32f7549fe72eb2b3919ca518d8cb3da62668"}
Aug 05 17:26:48.390209 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.390153    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"
Aug 05 17:26:48.390209 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.390176    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh" event={"ID":"c55c3126-506b-4819-b0d1-a90709270977","Type":"ContainerDied","Data":"8a77e9a7662ce48d9531b30b246dde6521ac8c57e074c42301349402d9e95c37"}
Aug 05 17:26:48.390359 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.390216    2574 scope.go:117] "RemoveContainer" containerID="ca289200789da3447a55f88cbcfaddcfa6bdcb799aec2f4fb5e6752e2598475b"
Aug 05 17:26:48.398521 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.398497    2574 scope.go:117] "RemoveContainer" containerID="6fe541318ad9653eb73494dafb1f32f7549fe72eb2b3919ca518d8cb3da62668"
Aug 05 17:26:48.406249 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.406225    2574 scope.go:117] "RemoveContainer" containerID="44482309e1a8b989839ada65c5b8249a7bb718e8856f1b1265112bf1ce5d99a8"
Aug 05 17:26:48.409499 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.409472    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"]
Aug 05 17:26:48.412223 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.412192    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"]
Aug 05 17:26:48.414526 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.414504    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5f455ffc7c-6spjh"]
Aug 05 17:26:48.416910 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.416892    2574 scope.go:117] "RemoveContainer" containerID="ca289200789da3447a55f88cbcfaddcfa6bdcb799aec2f4fb5e6752e2598475b"
Aug 05 17:26:48.417268 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:26:48.417244    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca289200789da3447a55f88cbcfaddcfa6bdcb799aec2f4fb5e6752e2598475b\": container with ID starting with ca289200789da3447a55f88cbcfaddcfa6bdcb799aec2f4fb5e6752e2598475b not found: ID does not exist" containerID="ca289200789da3447a55f88cbcfaddcfa6bdcb799aec2f4fb5e6752e2598475b"
Aug 05 17:26:48.417364 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.417284    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca289200789da3447a55f88cbcfaddcfa6bdcb799aec2f4fb5e6752e2598475b"} err="failed to get container status \"ca289200789da3447a55f88cbcfaddcfa6bdcb799aec2f4fb5e6752e2598475b\": rpc error: code = NotFound desc = could not find container \"ca289200789da3447a55f88cbcfaddcfa6bdcb799aec2f4fb5e6752e2598475b\": container with ID starting with ca289200789da3447a55f88cbcfaddcfa6bdcb799aec2f4fb5e6752e2598475b not found: ID does not exist"
Aug 05 17:26:48.417364 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.417314    2574 scope.go:117] "RemoveContainer" containerID="6fe541318ad9653eb73494dafb1f32f7549fe72eb2b3919ca518d8cb3da62668"
Aug 05 17:26:48.417625 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:26:48.417603    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe541318ad9653eb73494dafb1f32f7549fe72eb2b3919ca518d8cb3da62668\": container with ID starting with 6fe541318ad9653eb73494dafb1f32f7549fe72eb2b3919ca518d8cb3da62668 not found: ID does not exist" containerID="6fe541318ad9653eb73494dafb1f32f7549fe72eb2b3919ca518d8cb3da62668"
Aug 05 17:26:48.417705 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.417634    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe541318ad9653eb73494dafb1f32f7549fe72eb2b3919ca518d8cb3da62668"} err="failed to get container status \"6fe541318ad9653eb73494dafb1f32f7549fe72eb2b3919ca518d8cb3da62668\": rpc error: code = NotFound desc = could not find container \"6fe541318ad9653eb73494dafb1f32f7549fe72eb2b3919ca518d8cb3da62668\": container with ID starting with 6fe541318ad9653eb73494dafb1f32f7549fe72eb2b3919ca518d8cb3da62668 not found: ID does not exist"
Aug 05 17:26:48.417705 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.417659    2574 scope.go:117] "RemoveContainer" containerID="44482309e1a8b989839ada65c5b8249a7bb718e8856f1b1265112bf1ce5d99a8"
Aug 05 17:26:48.417817 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:26:48.417780    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode152081d_0127_44e1_907d_751da8caab98.slice/crio-fa447063c66b70138fecc6fb9a6436516baf79d64fb8dc9dd871ccf46ce3fda2 WatchSource:0}: Error finding container fa447063c66b70138fecc6fb9a6436516baf79d64fb8dc9dd871ccf46ce3fda2: Status 404 returned error can't find the container with id fa447063c66b70138fecc6fb9a6436516baf79d64fb8dc9dd871ccf46ce3fda2
Aug 05 17:26:48.417955 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:26:48.417934    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44482309e1a8b989839ada65c5b8249a7bb718e8856f1b1265112bf1ce5d99a8\": container with ID starting with 44482309e1a8b989839ada65c5b8249a7bb718e8856f1b1265112bf1ce5d99a8 not found: ID does not exist" containerID="44482309e1a8b989839ada65c5b8249a7bb718e8856f1b1265112bf1ce5d99a8"
Aug 05 17:26:48.418028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.417963    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44482309e1a8b989839ada65c5b8249a7bb718e8856f1b1265112bf1ce5d99a8"} err="failed to get container status \"44482309e1a8b989839ada65c5b8249a7bb718e8856f1b1265112bf1ce5d99a8\": rpc error: code = NotFound desc = could not find container \"44482309e1a8b989839ada65c5b8249a7bb718e8856f1b1265112bf1ce5d99a8\": container with ID starting with 44482309e1a8b989839ada65c5b8249a7bb718e8856f1b1265112bf1ce5d99a8 not found: ID does not exist"
Aug 05 17:26:48.525219 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:48.525183    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"]
Aug 05 17:26:48.529984 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:26:48.529957    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a40616_0eec_4d59_8776_95c7c986bb92.slice/crio-cb5edd1ded0f6ddea0cec929913e6541ee670a9846321d27eb1efb25439f5477 WatchSource:0}: Error finding container cb5edd1ded0f6ddea0cec929913e6541ee670a9846321d27eb1efb25439f5477: Status 404 returned error can't find the container with id cb5edd1ded0f6ddea0cec929913e6541ee670a9846321d27eb1efb25439f5477
Aug 05 17:26:49.396954 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:49.396917    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" event={"ID":"e152081d-0127-44e1-907d-751da8caab98","Type":"ContainerStarted","Data":"349e51838cb83cd4a1fda837a4ad110470998a3c8a965fc267e2261cdb43b6ab"}
Aug 05 17:26:49.397430 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:49.396973    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" event={"ID":"e152081d-0127-44e1-907d-751da8caab98","Type":"ContainerStarted","Data":"fa447063c66b70138fecc6fb9a6436516baf79d64fb8dc9dd871ccf46ce3fda2"}
Aug 05 17:26:49.398468 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:49.398447    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" event={"ID":"83a40616-0eec-4d59-8776-95c7c986bb92","Type":"ContainerStarted","Data":"3f2b63b7e0209b56447d39c09a97165e5717a2f07bfa8015f1e2f9b075c49ca8"}
Aug 05 17:26:49.398547 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:49.398496    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" event={"ID":"83a40616-0eec-4d59-8776-95c7c986bb92","Type":"ContainerStarted","Data":"cb5edd1ded0f6ddea0cec929913e6541ee670a9846321d27eb1efb25439f5477"}
Aug 05 17:26:50.068700 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:50.068658    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Aug 05 17:26:50.195564 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:50.195526    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c55c3126-506b-4819-b0d1-a90709270977" path="/var/lib/kubelet/pods/c55c3126-506b-4819-b0d1-a90709270977/volumes"
Aug 05 17:26:53.416435 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:53.416401    2574 generic.go:356] "Generic (PLEG): container finished" podID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerID="56662a61bc587fce9ccff2e79f9d0f768b0c6f8a4a8df2136027c0fd20b12451" exitCode=0
Aug 05 17:26:53.416868 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:53.416478    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" event={"ID":"64d21b6a-55de-40b5-b65f-0acca1fc29b9","Type":"ContainerDied","Data":"56662a61bc587fce9ccff2e79f9d0f768b0c6f8a4a8df2136027c0fd20b12451"}
Aug 05 17:26:53.417869 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:53.417845    2574 generic.go:356] "Generic (PLEG): container finished" podID="e152081d-0127-44e1-907d-751da8caab98" containerID="349e51838cb83cd4a1fda837a4ad110470998a3c8a965fc267e2261cdb43b6ab" exitCode=0
Aug 05 17:26:53.417968 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:53.417927    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" event={"ID":"e152081d-0127-44e1-907d-751da8caab98","Type":"ContainerDied","Data":"349e51838cb83cd4a1fda837a4ad110470998a3c8a965fc267e2261cdb43b6ab"}
Aug 05 17:26:53.419384 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:53.419364    2574 generic.go:356] "Generic (PLEG): container finished" podID="83a40616-0eec-4d59-8776-95c7c986bb92" containerID="3f2b63b7e0209b56447d39c09a97165e5717a2f07bfa8015f1e2f9b075c49ca8" exitCode=0
Aug 05 17:26:53.419460 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:53.419387    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" event={"ID":"83a40616-0eec-4d59-8776-95c7c986bb92","Type":"ContainerDied","Data":"3f2b63b7e0209b56447d39c09a97165e5717a2f07bfa8015f1e2f9b075c49ca8"}
Aug 05 17:26:53.420608 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:53.420594    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 05 17:26:54.427848 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:54.427805    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" event={"ID":"e152081d-0127-44e1-907d-751da8caab98","Type":"ContainerStarted","Data":"e9d4e03b0e421cc8b9e6b58cdf968a4b3faf6234f48ff8fb1abfd69076ae0f1e"}
Aug 05 17:26:54.428340 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:54.427887    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" event={"ID":"e152081d-0127-44e1-907d-751da8caab98","Type":"ContainerStarted","Data":"c88ba8257ddfef7994ce9a5542af4df1a76adedf564693df36b2d721ece3171f"}
Aug 05 17:26:54.428340 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:54.428140    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:54.449755 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:54.449698    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" podStartSLOduration=3.5256320370000003 podStartE2EDuration="7.449677989s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.924045952s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:26:54.448418699 +0000 UTC m=+864.740273550" watchObservedRunningTime="2026-08-05 17:26:54.449677989 +0000 UTC m=+864.741532820"
Aug 05 17:26:55.068859 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:55.068813    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Aug 05 17:26:55.433952 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:55.433860    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:26:55.435275 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:55.435232    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Aug 05 17:26:56.438196 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:56.438139    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Aug 05 17:26:57.076086 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:57.076037    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Aug 05 17:26:57.076401 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:26:57.076372    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:27:00.068692 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:00.068640    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Aug 05 17:27:00.069185 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:00.068773    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:27:01.443556 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:01.443527    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:27:01.444187 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:01.444158    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Aug 05 17:27:05.068684 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:05.068639    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Aug 05 17:27:07.076617 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:07.076548    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Aug 05 17:27:07.077064 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:07.076883    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:27:10.068839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:10.068656    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Aug 05 17:27:11.445049 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:11.445007    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Aug 05 17:27:13.517033 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:13.516997    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" event={"ID":"83a40616-0eec-4d59-8776-95c7c986bb92","Type":"ContainerStarted","Data":"4a8cfb89cd93a3d09c390072a51b3707b675acb602072ba21537ce187d797bd0"}
Aug 05 17:27:13.517544 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:13.517098    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" event={"ID":"83a40616-0eec-4d59-8776-95c7c986bb92","Type":"ContainerStarted","Data":"a3623bd5791f2d290908c7e0effd4382dca58fe3eaff0da20e54601547c8b792"}
Aug 05 17:27:13.517544 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:13.517383    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:27:13.517544 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:13.517441    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:27:13.518544 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:13.518516    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 05 17:27:13.540200 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:13.540149    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" podStartSLOduration=2.113060483 podStartE2EDuration="25.540130179s" totalImagesPullingTime="19.525487631s" totalInitContainerRuntime="3.901582065s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:27:13.53979359 +0000 UTC m=+883.831648434" watchObservedRunningTime="2026-08-05 17:27:13.540130179 +0000 UTC m=+883.831985011"
Aug 05 17:27:14.523114 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:14.523048    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 05 17:27:15.068720 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:15.068671    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Aug 05 17:27:17.076851 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:17.076798    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Aug 05 17:27:17.077298 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:17.076975    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:27:17.077298 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:17.077111    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:27:17.077298 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:17.077223    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:27:18.535378 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.535349    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:27:18.541826 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.541799    2574 generic.go:356] "Generic (PLEG): container finished" podID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerID="258004ad4e5cc26ceae581c281f9f188af60bd60625eb4806b884360192c6477" exitCode=0
Aug 05 17:27:18.541970 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.541849    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" event={"ID":"64d21b6a-55de-40b5-b65f-0acca1fc29b9","Type":"ContainerDied","Data":"258004ad4e5cc26ceae581c281f9f188af60bd60625eb4806b884360192c6477"}
Aug 05 17:27:18.541970 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.541866    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"
Aug 05 17:27:18.541970 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.541901    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk" event={"ID":"64d21b6a-55de-40b5-b65f-0acca1fc29b9","Type":"ContainerDied","Data":"457678e33d3487cd3738d282588a5e1e5f62286f8345e06672a5bf4603db4027"}
Aug 05 17:27:18.541970 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.541933    2574 scope.go:117] "RemoveContainer" containerID="258004ad4e5cc26ceae581c281f9f188af60bd60625eb4806b884360192c6477"
Aug 05 17:27:18.549542 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.549525    2574 scope.go:117] "RemoveContainer" containerID="b2a5da8da8234ad63c3592fcd4be805b70a90f2ae2478de692326b6daac8b50d"
Aug 05 17:27:18.556549 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.556530    2574 scope.go:117] "RemoveContainer" containerID="56662a61bc587fce9ccff2e79f9d0f768b0c6f8a4a8df2136027c0fd20b12451"
Aug 05 17:27:18.565989 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.565967    2574 scope.go:117] "RemoveContainer" containerID="2f322729ce85b12fa6dedd11a0ca1a97ed23c3b4685897d3dbcf664146c44cca"
Aug 05 17:27:18.573344 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.573321    2574 scope.go:117] "RemoveContainer" containerID="258004ad4e5cc26ceae581c281f9f188af60bd60625eb4806b884360192c6477"
Aug 05 17:27:18.573611 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:27:18.573591    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258004ad4e5cc26ceae581c281f9f188af60bd60625eb4806b884360192c6477\": container with ID starting with 258004ad4e5cc26ceae581c281f9f188af60bd60625eb4806b884360192c6477 not found: ID does not exist" containerID="258004ad4e5cc26ceae581c281f9f188af60bd60625eb4806b884360192c6477"
Aug 05 17:27:18.573682 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.573619    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258004ad4e5cc26ceae581c281f9f188af60bd60625eb4806b884360192c6477"} err="failed to get container status \"258004ad4e5cc26ceae581c281f9f188af60bd60625eb4806b884360192c6477\": rpc error: code = NotFound desc = could not find container \"258004ad4e5cc26ceae581c281f9f188af60bd60625eb4806b884360192c6477\": container with ID starting with 258004ad4e5cc26ceae581c281f9f188af60bd60625eb4806b884360192c6477 not found: ID does not exist"
Aug 05 17:27:18.573682 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.573639    2574 scope.go:117] "RemoveContainer" containerID="b2a5da8da8234ad63c3592fcd4be805b70a90f2ae2478de692326b6daac8b50d"
Aug 05 17:27:18.573885 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:27:18.573863    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a5da8da8234ad63c3592fcd4be805b70a90f2ae2478de692326b6daac8b50d\": container with ID starting with b2a5da8da8234ad63c3592fcd4be805b70a90f2ae2478de692326b6daac8b50d not found: ID does not exist" containerID="b2a5da8da8234ad63c3592fcd4be805b70a90f2ae2478de692326b6daac8b50d"
Aug 05 17:27:18.573957 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.573892    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a5da8da8234ad63c3592fcd4be805b70a90f2ae2478de692326b6daac8b50d"} err="failed to get container status \"b2a5da8da8234ad63c3592fcd4be805b70a90f2ae2478de692326b6daac8b50d\": rpc error: code = NotFound desc = could not find container \"b2a5da8da8234ad63c3592fcd4be805b70a90f2ae2478de692326b6daac8b50d\": container with ID starting with b2a5da8da8234ad63c3592fcd4be805b70a90f2ae2478de692326b6daac8b50d not found: ID does not exist"
Aug 05 17:27:18.573957 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.573909    2574 scope.go:117] "RemoveContainer" containerID="56662a61bc587fce9ccff2e79f9d0f768b0c6f8a4a8df2136027c0fd20b12451"
Aug 05 17:27:18.574205 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:27:18.574186    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56662a61bc587fce9ccff2e79f9d0f768b0c6f8a4a8df2136027c0fd20b12451\": container with ID starting with 56662a61bc587fce9ccff2e79f9d0f768b0c6f8a4a8df2136027c0fd20b12451 not found: ID does not exist" containerID="56662a61bc587fce9ccff2e79f9d0f768b0c6f8a4a8df2136027c0fd20b12451"
Aug 05 17:27:18.574280 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.574209    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56662a61bc587fce9ccff2e79f9d0f768b0c6f8a4a8df2136027c0fd20b12451"} err="failed to get container status \"56662a61bc587fce9ccff2e79f9d0f768b0c6f8a4a8df2136027c0fd20b12451\": rpc error: code = NotFound desc = could not find container \"56662a61bc587fce9ccff2e79f9d0f768b0c6f8a4a8df2136027c0fd20b12451\": container with ID starting with 56662a61bc587fce9ccff2e79f9d0f768b0c6f8a4a8df2136027c0fd20b12451 not found: ID does not exist"
Aug 05 17:27:18.574280 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.574224    2574 scope.go:117] "RemoveContainer" containerID="2f322729ce85b12fa6dedd11a0ca1a97ed23c3b4685897d3dbcf664146c44cca"
Aug 05 17:27:18.574476 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:27:18.574456    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f322729ce85b12fa6dedd11a0ca1a97ed23c3b4685897d3dbcf664146c44cca\": container with ID starting with 2f322729ce85b12fa6dedd11a0ca1a97ed23c3b4685897d3dbcf664146c44cca not found: ID does not exist" containerID="2f322729ce85b12fa6dedd11a0ca1a97ed23c3b4685897d3dbcf664146c44cca"
Aug 05 17:27:18.574541 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.574478    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f322729ce85b12fa6dedd11a0ca1a97ed23c3b4685897d3dbcf664146c44cca"} err="failed to get container status \"2f322729ce85b12fa6dedd11a0ca1a97ed23c3b4685897d3dbcf664146c44cca\": rpc error: code = NotFound desc = could not find container \"2f322729ce85b12fa6dedd11a0ca1a97ed23c3b4685897d3dbcf664146c44cca\": container with ID starting with 2f322729ce85b12fa6dedd11a0ca1a97ed23c3b4685897d3dbcf664146c44cca not found: ID does not exist"
Aug 05 17:27:18.657754 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.657660    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kube-rbac-proxy-sar-config\") pod \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") "
Aug 05 17:27:18.657754 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.657701    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kserve-provision-location\") pod \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") "
Aug 05 17:27:18.657754 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.657739    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zjqb\" (UniqueName: \"kubernetes.io/projected/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kube-api-access-5zjqb\") pod \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") "
Aug 05 17:27:18.658044 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.657765    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64d21b6a-55de-40b5-b65f-0acca1fc29b9-cabundle-cert\") pod \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") "
Aug 05 17:27:18.658044 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.657787    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64d21b6a-55de-40b5-b65f-0acca1fc29b9-proxy-tls\") pod \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\" (UID: \"64d21b6a-55de-40b5-b65f-0acca1fc29b9\") "
Aug 05 17:27:18.658177 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.658068    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64d21b6a-55de-40b5-b65f-0acca1fc29b9" (UID: "64d21b6a-55de-40b5-b65f-0acca1fc29b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:27:18.658177 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.658151    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d21b6a-55de-40b5-b65f-0acca1fc29b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64d21b6a-55de-40b5-b65f-0acca1fc29b9" (UID: "64d21b6a-55de-40b5-b65f-0acca1fc29b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:27:18.658177 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.658148    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "64d21b6a-55de-40b5-b65f-0acca1fc29b9" (UID: "64d21b6a-55de-40b5-b65f-0acca1fc29b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:27:18.659906 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.659885    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d21b6a-55de-40b5-b65f-0acca1fc29b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "64d21b6a-55de-40b5-b65f-0acca1fc29b9" (UID: "64d21b6a-55de-40b5-b65f-0acca1fc29b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:27:18.659959 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.659904    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kube-api-access-5zjqb" (OuterVolumeSpecName: "kube-api-access-5zjqb") pod "64d21b6a-55de-40b5-b65f-0acca1fc29b9" (UID: "64d21b6a-55de-40b5-b65f-0acca1fc29b9"). InnerVolumeSpecName "kube-api-access-5zjqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:27:18.758483 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.758443    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zjqb\" (UniqueName: \"kubernetes.io/projected/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kube-api-access-5zjqb\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:27:18.758483 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.758477    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64d21b6a-55de-40b5-b65f-0acca1fc29b9-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:27:18.758483 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.758486    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64d21b6a-55de-40b5-b65f-0acca1fc29b9-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:27:18.758483 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.758495    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:27:18.758751 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.758504    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64d21b6a-55de-40b5-b65f-0acca1fc29b9-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:27:18.865486 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.865452    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"]
Aug 05 17:27:18.870923 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:18.870893    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-fdd1f-predictor-77bff7fbbf-s2rlk"]
Aug 05 17:27:19.528559 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:19.528532    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:27:19.529097 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:19.529059    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 05 17:27:20.193609 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:20.193577    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" path="/var/lib/kubelet/pods/64d21b6a-55de-40b5-b65f-0acca1fc29b9/volumes"
Aug 05 17:27:21.444765 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:21.444723    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Aug 05 17:27:29.529388 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:29.529344    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 05 17:27:31.444979 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:31.444935    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Aug 05 17:27:39.529701 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:39.529657    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 05 17:27:41.444157 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:41.444120    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Aug 05 17:27:49.529207 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:49.529166    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 05 17:27:51.444876 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:51.444832    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Aug 05 17:27:59.529205 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:27:59.529166    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 05 17:28:01.444595 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:01.444554    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Aug 05 17:28:09.529937 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:09.529896    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 05 17:28:11.444861 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:11.444831    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:28:19.530341 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:19.530305    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:28:32.958213 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:32.958178    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"]
Aug 05 17:28:32.958825 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:32.958468    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kserve-container" containerID="cri-o://2f51df0780d90bc4663cbb76754dc9bcd3e6f7727e3bac6e87d87aff519fa202" gracePeriod=30
Aug 05 17:28:32.958825 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:32.958494    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kube-rbac-proxy" containerID="cri-o://c51584f79ba2898f8c0fdb7e880ff148a80bbb1eb7b18380f9173bf0a266edeb" gracePeriod=30
Aug 05 17:28:33.057323 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:33.057279    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Aug 05 17:28:33.822797 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:33.822761    2574 generic.go:356] "Generic (PLEG): container finished" podID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerID="c51584f79ba2898f8c0fdb7e880ff148a80bbb1eb7b18380f9173bf0a266edeb" exitCode=2
Aug 05 17:28:33.822984 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:33.822839    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" event={"ID":"bba18d24-a734-4f0f-80ce-e78439c474c7","Type":"ContainerDied","Data":"c51584f79ba2898f8c0fdb7e880ff148a80bbb1eb7b18380f9173bf0a266edeb"}
Aug 05 17:28:37.797758 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.797734    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:28:37.840364 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.840332    2574 generic.go:356] "Generic (PLEG): container finished" podID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerID="2f51df0780d90bc4663cbb76754dc9bcd3e6f7727e3bac6e87d87aff519fa202" exitCode=0
Aug 05 17:28:37.840554 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.840426    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"
Aug 05 17:28:37.840554 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.840427    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" event={"ID":"bba18d24-a734-4f0f-80ce-e78439c474c7","Type":"ContainerDied","Data":"2f51df0780d90bc4663cbb76754dc9bcd3e6f7727e3bac6e87d87aff519fa202"}
Aug 05 17:28:37.840554 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.840506    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m" event={"ID":"bba18d24-a734-4f0f-80ce-e78439c474c7","Type":"ContainerDied","Data":"7a1875c75916ea1ec7465684a858d3441b41606a058e632063ea640136d6ebfa"}
Aug 05 17:28:37.840554 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.840541    2574 scope.go:117] "RemoveContainer" containerID="c51584f79ba2898f8c0fdb7e880ff148a80bbb1eb7b18380f9173bf0a266edeb"
Aug 05 17:28:37.848391 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.848372    2574 scope.go:117] "RemoveContainer" containerID="2f51df0780d90bc4663cbb76754dc9bcd3e6f7727e3bac6e87d87aff519fa202"
Aug 05 17:28:37.855576 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.855558    2574 scope.go:117] "RemoveContainer" containerID="97f278efb2e756f54bf48830778722622be5703ae800c733e996c70961e414c0"
Aug 05 17:28:37.862507 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.862489    2574 scope.go:117] "RemoveContainer" containerID="c51584f79ba2898f8c0fdb7e880ff148a80bbb1eb7b18380f9173bf0a266edeb"
Aug 05 17:28:37.862754 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:28:37.862734    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51584f79ba2898f8c0fdb7e880ff148a80bbb1eb7b18380f9173bf0a266edeb\": container with ID starting with c51584f79ba2898f8c0fdb7e880ff148a80bbb1eb7b18380f9173bf0a266edeb not found: ID does not exist" containerID="c51584f79ba2898f8c0fdb7e880ff148a80bbb1eb7b18380f9173bf0a266edeb"
Aug 05 17:28:37.862801 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.862763    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51584f79ba2898f8c0fdb7e880ff148a80bbb1eb7b18380f9173bf0a266edeb"} err="failed to get container status \"c51584f79ba2898f8c0fdb7e880ff148a80bbb1eb7b18380f9173bf0a266edeb\": rpc error: code = NotFound desc = could not find container \"c51584f79ba2898f8c0fdb7e880ff148a80bbb1eb7b18380f9173bf0a266edeb\": container with ID starting with c51584f79ba2898f8c0fdb7e880ff148a80bbb1eb7b18380f9173bf0a266edeb not found: ID does not exist"
Aug 05 17:28:37.862801 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.862783    2574 scope.go:117] "RemoveContainer" containerID="2f51df0780d90bc4663cbb76754dc9bcd3e6f7727e3bac6e87d87aff519fa202"
Aug 05 17:28:37.863044 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:28:37.863027    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f51df0780d90bc4663cbb76754dc9bcd3e6f7727e3bac6e87d87aff519fa202\": container with ID starting with 2f51df0780d90bc4663cbb76754dc9bcd3e6f7727e3bac6e87d87aff519fa202 not found: ID does not exist" containerID="2f51df0780d90bc4663cbb76754dc9bcd3e6f7727e3bac6e87d87aff519fa202"
Aug 05 17:28:37.863178 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.863053    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f51df0780d90bc4663cbb76754dc9bcd3e6f7727e3bac6e87d87aff519fa202"} err="failed to get container status \"2f51df0780d90bc4663cbb76754dc9bcd3e6f7727e3bac6e87d87aff519fa202\": rpc error: code = NotFound desc = could not find container \"2f51df0780d90bc4663cbb76754dc9bcd3e6f7727e3bac6e87d87aff519fa202\": container with ID starting with 2f51df0780d90bc4663cbb76754dc9bcd3e6f7727e3bac6e87d87aff519fa202 not found: ID does not exist"
Aug 05 17:28:37.863223 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.863180    2574 scope.go:117] "RemoveContainer" containerID="97f278efb2e756f54bf48830778722622be5703ae800c733e996c70961e414c0"
Aug 05 17:28:37.863433 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:28:37.863418    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f278efb2e756f54bf48830778722622be5703ae800c733e996c70961e414c0\": container with ID starting with 97f278efb2e756f54bf48830778722622be5703ae800c733e996c70961e414c0 not found: ID does not exist" containerID="97f278efb2e756f54bf48830778722622be5703ae800c733e996c70961e414c0"
Aug 05 17:28:37.863478 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.863437    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f278efb2e756f54bf48830778722622be5703ae800c733e996c70961e414c0"} err="failed to get container status \"97f278efb2e756f54bf48830778722622be5703ae800c733e996c70961e414c0\": rpc error: code = NotFound desc = could not find container \"97f278efb2e756f54bf48830778722622be5703ae800c733e996c70961e414c0\": container with ID starting with 97f278efb2e756f54bf48830778722622be5703ae800c733e996c70961e414c0 not found: ID does not exist"
Aug 05 17:28:37.872773 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.872717    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bba18d24-a734-4f0f-80ce-e78439c474c7-cabundle-cert\") pod \"bba18d24-a734-4f0f-80ce-e78439c474c7\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") "
Aug 05 17:28:37.872773 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.872758    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bba18d24-a734-4f0f-80ce-e78439c474c7-kserve-provision-location\") pod \"bba18d24-a734-4f0f-80ce-e78439c474c7\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") "
Aug 05 17:28:37.872986 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.872819    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfg67\" (UniqueName: \"kubernetes.io/projected/bba18d24-a734-4f0f-80ce-e78439c474c7-kube-api-access-lfg67\") pod \"bba18d24-a734-4f0f-80ce-e78439c474c7\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") "
Aug 05 17:28:37.872986 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.872854    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bba18d24-a734-4f0f-80ce-e78439c474c7-kube-rbac-proxy-sar-config\") pod \"bba18d24-a734-4f0f-80ce-e78439c474c7\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") "
Aug 05 17:28:37.872986 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.872911    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bba18d24-a734-4f0f-80ce-e78439c474c7-proxy-tls\") pod \"bba18d24-a734-4f0f-80ce-e78439c474c7\" (UID: \"bba18d24-a734-4f0f-80ce-e78439c474c7\") "
Aug 05 17:28:37.873153 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.873126    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba18d24-a734-4f0f-80ce-e78439c474c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bba18d24-a734-4f0f-80ce-e78439c474c7" (UID: "bba18d24-a734-4f0f-80ce-e78439c474c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:28:37.873190 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.873146    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba18d24-a734-4f0f-80ce-e78439c474c7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bba18d24-a734-4f0f-80ce-e78439c474c7" (UID: "bba18d24-a734-4f0f-80ce-e78439c474c7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:28:37.873264 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.873243    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba18d24-a734-4f0f-80ce-e78439c474c7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bba18d24-a734-4f0f-80ce-e78439c474c7" (UID: "bba18d24-a734-4f0f-80ce-e78439c474c7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:28:37.874876 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.874855    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bba18d24-a734-4f0f-80ce-e78439c474c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bba18d24-a734-4f0f-80ce-e78439c474c7" (UID: "bba18d24-a734-4f0f-80ce-e78439c474c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:28:37.874951 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.874930    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba18d24-a734-4f0f-80ce-e78439c474c7-kube-api-access-lfg67" (OuterVolumeSpecName: "kube-api-access-lfg67") pod "bba18d24-a734-4f0f-80ce-e78439c474c7" (UID: "bba18d24-a734-4f0f-80ce-e78439c474c7"). InnerVolumeSpecName "kube-api-access-lfg67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:28:37.973944 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.973910    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bba18d24-a734-4f0f-80ce-e78439c474c7-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:28:37.973944 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.973937    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bba18d24-a734-4f0f-80ce-e78439c474c7-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:28:37.973944 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.973946    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bba18d24-a734-4f0f-80ce-e78439c474c7-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:28:37.974212 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.973957    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfg67\" (UniqueName: \"kubernetes.io/projected/bba18d24-a734-4f0f-80ce-e78439c474c7-kube-api-access-lfg67\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:28:37.974212 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:37.973968    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bba18d24-a734-4f0f-80ce-e78439c474c7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:28:38.150381 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.150291    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"]
Aug 05 17:28:38.151193 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.151168    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kserve-container"
Aug 05 17:28:38.151349 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.151336    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kserve-container"
Aug 05 17:28:38.151462 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.151448    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="storage-initializer"
Aug 05 17:28:38.151559 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.151545    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="storage-initializer"
Aug 05 17:28:38.151651 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.151639    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent"
Aug 05 17:28:38.151728 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.151718    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent"
Aug 05 17:28:38.151883 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.151866    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kube-rbac-proxy"
Aug 05 17:28:38.151975 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.151966    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kube-rbac-proxy"
Aug 05 17:28:38.152737 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.152721    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kube-rbac-proxy"
Aug 05 17:28:38.153945 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.153880    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kube-rbac-proxy"
Aug 05 17:28:38.154030 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.153969    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="storage-initializer"
Aug 05 17:28:38.154030 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.153985    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="storage-initializer"
Aug 05 17:28:38.154030 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.154011    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container"
Aug 05 17:28:38.154030 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.154021    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container"
Aug 05 17:28:38.154317 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.154177    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kube-rbac-proxy"
Aug 05 17:28:38.154317 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.154191    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kserve-container"
Aug 05 17:28:38.154317 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.154202    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="kserve-container"
Aug 05 17:28:38.154317 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.154219    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" containerName="kube-rbac-proxy"
Aug 05 17:28:38.154317 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.154227    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="64d21b6a-55de-40b5-b65f-0acca1fc29b9" containerName="agent"
Aug 05 17:28:38.158506 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.158484    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.161092 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.161048    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Aug 05 17:28:38.161397 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.161047    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"]
Aug 05 17:28:38.161576 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.161064    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Aug 05 17:28:38.175271 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.175248    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.175379 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.175289    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxlxs\" (UniqueName: \"kubernetes.io/projected/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kube-api-access-nxlxs\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.175379 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.175318    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.175379 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.175338    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.175379 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.175365    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.178599 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.178573    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"]
Aug 05 17:28:38.180188 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:28:38.180162    2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba18d24_a734_4f0f_80ce_e78439c474c7.slice\": RecentStats: unable to find data in memory cache]"
Aug 05 17:28:38.184703 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.184676    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-55cddd885b-kmz8m"]
Aug 05 17:28:38.193665 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.193631    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bba18d24-a734-4f0f-80ce-e78439c474c7" path="/var/lib/kubelet/pods/bba18d24-a734-4f0f-80ce-e78439c474c7/volumes"
Aug 05 17:28:38.276588 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.276551    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.276778 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.276643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxlxs\" (UniqueName: \"kubernetes.io/projected/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kube-api-access-nxlxs\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.276778 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.276676    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.276778 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.276700    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.276778 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.276739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.277216 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:28:38.277195    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Aug 05 17:28:38.277339 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:28:38.277324    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-proxy-tls podName:0d23bb97-f7a8-4727-90f1-06f9fd246e9b nodeName:}" failed. No retries permitted until 2026-08-05 17:28:38.777304034 +0000 UTC m=+969.069158843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-proxy-tls") pod "isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" (UID: "0d23bb97-f7a8-4727-90f1-06f9fd246e9b") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Aug 05 17:28:38.277440 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.277229    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.277873 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.277852    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.277873 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.277870    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.286147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.286118    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxlxs\" (UniqueName: \"kubernetes.io/projected/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kube-api-access-nxlxs\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.294616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.294585    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"]
Aug 05 17:28:38.294888 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.294862    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kserve-container" containerID="cri-o://c88ba8257ddfef7994ce9a5542af4df1a76adedf564693df36b2d721ece3171f" gracePeriod=30
Aug 05 17:28:38.294987 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.294956    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kube-rbac-proxy" containerID="cri-o://e9d4e03b0e421cc8b9e6b58cdf968a4b3faf6234f48ff8fb1abfd69076ae0f1e" gracePeriod=30
Aug 05 17:28:38.421306 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.421274    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"]
Aug 05 17:28:38.421550 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.421526    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kserve-container" containerID="cri-o://a3623bd5791f2d290908c7e0effd4382dca58fe3eaff0da20e54601547c8b792" gracePeriod=30
Aug 05 17:28:38.421622 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.421560    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kube-rbac-proxy" containerID="cri-o://4a8cfb89cd93a3d09c390072a51b3707b675acb602072ba21537ce187d797bd0" gracePeriod=30
Aug 05 17:28:38.538573 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.538540    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"]
Aug 05 17:28:38.541121 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.541101    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:38.543423 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.543400    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-f37da-kube-rbac-proxy-sar-config\""
Aug 05 17:28:38.543423 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.543401    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-f37da-predictor-serving-cert\""
Aug 05 17:28:38.553022 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.552996    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"]
Aug 05 17:28:38.578749 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.578707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/583d6899-27a8-4b78-90d1-8c7c871a3c4c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:38.578942 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.578858    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/583d6899-27a8-4b78-90d1-8c7c871a3c4c-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:38.578942 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.578923    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjgsb\" (UniqueName: \"kubernetes.io/projected/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kube-api-access-jjgsb\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:38.579058 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.578961    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:38.579058 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.579017    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:38.679535 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.679438    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjgsb\" (UniqueName: \"kubernetes.io/projected/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kube-api-access-jjgsb\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:38.679535 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.679487    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:38.679535 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.679516    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:38.679819 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.679552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/583d6899-27a8-4b78-90d1-8c7c871a3c4c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:38.679819 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.679640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/583d6899-27a8-4b78-90d1-8c7c871a3c4c-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:38.679819 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:28:38.679770    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-f37da-predictor-serving-cert" not found
Aug 05 17:28:38.679972 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:28:38.679833    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/583d6899-27a8-4b78-90d1-8c7c871a3c4c-proxy-tls podName:583d6899-27a8-4b78-90d1-8c7c871a3c4c nodeName:}" failed. No retries permitted until 2026-08-05 17:28:39.179812736 +0000 UTC m=+969.471667559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/583d6899-27a8-4b78-90d1-8c7c871a3c4c-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" (UID: "583d6899-27a8-4b78-90d1-8c7c871a3c4c") : secret "isvc-xgboost-graph-raw-hpa-f37da-predictor-serving-cert" not found
Aug 05 17:28:38.679972 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.679927    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:38.680254 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.680237    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/583d6899-27a8-4b78-90d1-8c7c871a3c4c-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:38.680303 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.680267    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:38.688139 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.688107    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjgsb\" (UniqueName: \"kubernetes.io/projected/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kube-api-access-jjgsb\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:38.780133 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.780095    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.782664 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.782640    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-779884cd8f-zpgkf\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:38.846695 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.846650    2574 generic.go:356] "Generic (PLEG): container finished" podID="e152081d-0127-44e1-907d-751da8caab98" containerID="e9d4e03b0e421cc8b9e6b58cdf968a4b3faf6234f48ff8fb1abfd69076ae0f1e" exitCode=2
Aug 05 17:28:38.847110 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.846726    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" event={"ID":"e152081d-0127-44e1-907d-751da8caab98","Type":"ContainerDied","Data":"e9d4e03b0e421cc8b9e6b58cdf968a4b3faf6234f48ff8fb1abfd69076ae0f1e"}
Aug 05 17:28:38.848718 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.848694    2574 generic.go:356] "Generic (PLEG): container finished" podID="83a40616-0eec-4d59-8776-95c7c986bb92" containerID="4a8cfb89cd93a3d09c390072a51b3707b675acb602072ba21537ce187d797bd0" exitCode=2
Aug 05 17:28:38.848833 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:38.848768    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" event={"ID":"83a40616-0eec-4d59-8776-95c7c986bb92","Type":"ContainerDied","Data":"4a8cfb89cd93a3d09c390072a51b3707b675acb602072ba21537ce187d797bd0"}
Aug 05 17:28:39.071906 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:39.071872    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:39.182606 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:39.182575    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/583d6899-27a8-4b78-90d1-8c7c871a3c4c-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:39.185282 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:39.185251    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/583d6899-27a8-4b78-90d1-8c7c871a3c4c-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:39.205391 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:39.205366    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"]
Aug 05 17:28:39.207384 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:28:39.207359    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d23bb97_f7a8_4727_90f1_06f9fd246e9b.slice/crio-146b7086e056687a060b0c252f8e663790273b47b33b2338bd798fcfb81f6de0 WatchSource:0}: Error finding container 146b7086e056687a060b0c252f8e663790273b47b33b2338bd798fcfb81f6de0: Status 404 returned error can't find the container with id 146b7086e056687a060b0c252f8e663790273b47b33b2338bd798fcfb81f6de0
Aug 05 17:28:39.451644 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:39.451555    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:39.523808 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:39.523765    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Aug 05 17:28:39.529236 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:39.529202    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 05 17:28:39.592060 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:39.592025    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"]
Aug 05 17:28:39.595570 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:28:39.595541    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583d6899_27a8_4b78_90d1_8c7c871a3c4c.slice/crio-3e36e23abc557e2af511b55db166eb6e96ae7d09f2eb0123c1b1802641bc1a27 WatchSource:0}: Error finding container 3e36e23abc557e2af511b55db166eb6e96ae7d09f2eb0123c1b1802641bc1a27: Status 404 returned error can't find the container with id 3e36e23abc557e2af511b55db166eb6e96ae7d09f2eb0123c1b1802641bc1a27
Aug 05 17:28:39.855644 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:39.855421    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" event={"ID":"0d23bb97-f7a8-4727-90f1-06f9fd246e9b","Type":"ContainerStarted","Data":"c78732b5f79779250e54d43a1284e30608aa424f0a640c390c5b918548d78b30"}
Aug 05 17:28:39.855644 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:39.855502    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" event={"ID":"0d23bb97-f7a8-4727-90f1-06f9fd246e9b","Type":"ContainerStarted","Data":"146b7086e056687a060b0c252f8e663790273b47b33b2338bd798fcfb81f6de0"}
Aug 05 17:28:39.857213 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:39.857185    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" event={"ID":"583d6899-27a8-4b78-90d1-8c7c871a3c4c","Type":"ContainerStarted","Data":"655804b7e52effc277b6f658d28e7d6cea0edb0a4a53d9823db4b58c10d32a26"}
Aug 05 17:28:39.857331 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:39.857234    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" event={"ID":"583d6899-27a8-4b78-90d1-8c7c871a3c4c","Type":"ContainerStarted","Data":"3e36e23abc557e2af511b55db166eb6e96ae7d09f2eb0123c1b1802641bc1a27"}
Aug 05 17:28:41.439317 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:41.439271    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Aug 05 17:28:41.444703 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:41.444675    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Aug 05 17:28:42.367543 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.367512    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:28:42.404974 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.404943    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83a40616-0eec-4d59-8776-95c7c986bb92-kube-rbac-proxy-sar-config\") pod \"83a40616-0eec-4d59-8776-95c7c986bb92\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") "
Aug 05 17:28:42.405158 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.404996    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx6vl\" (UniqueName: \"kubernetes.io/projected/83a40616-0eec-4d59-8776-95c7c986bb92-kube-api-access-gx6vl\") pod \"83a40616-0eec-4d59-8776-95c7c986bb92\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") "
Aug 05 17:28:42.405158 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.405064    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83a40616-0eec-4d59-8776-95c7c986bb92-cabundle-cert\") pod \"83a40616-0eec-4d59-8776-95c7c986bb92\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") "
Aug 05 17:28:42.405158 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.405108    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83a40616-0eec-4d59-8776-95c7c986bb92-kserve-provision-location\") pod \"83a40616-0eec-4d59-8776-95c7c986bb92\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") "
Aug 05 17:28:42.405158 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.405149    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83a40616-0eec-4d59-8776-95c7c986bb92-proxy-tls\") pod \"83a40616-0eec-4d59-8776-95c7c986bb92\" (UID: \"83a40616-0eec-4d59-8776-95c7c986bb92\") "
Aug 05 17:28:42.405436 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.405407    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a40616-0eec-4d59-8776-95c7c986bb92-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "83a40616-0eec-4d59-8776-95c7c986bb92" (UID: "83a40616-0eec-4d59-8776-95c7c986bb92"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:28:42.405516 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.405445    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a40616-0eec-4d59-8776-95c7c986bb92-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "83a40616-0eec-4d59-8776-95c7c986bb92" (UID: "83a40616-0eec-4d59-8776-95c7c986bb92"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:28:42.405610 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.405588    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a40616-0eec-4d59-8776-95c7c986bb92-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83a40616-0eec-4d59-8776-95c7c986bb92" (UID: "83a40616-0eec-4d59-8776-95c7c986bb92"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:28:42.407270 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.407242    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a40616-0eec-4d59-8776-95c7c986bb92-kube-api-access-gx6vl" (OuterVolumeSpecName: "kube-api-access-gx6vl") pod "83a40616-0eec-4d59-8776-95c7c986bb92" (UID: "83a40616-0eec-4d59-8776-95c7c986bb92"). InnerVolumeSpecName "kube-api-access-gx6vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:28:42.407357 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.407283    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a40616-0eec-4d59-8776-95c7c986bb92-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83a40616-0eec-4d59-8776-95c7c986bb92" (UID: "83a40616-0eec-4d59-8776-95c7c986bb92"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:28:42.505708 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.505668    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83a40616-0eec-4d59-8776-95c7c986bb92-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:28:42.505708 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.505703    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83a40616-0eec-4d59-8776-95c7c986bb92-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:28:42.505708 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.505715    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83a40616-0eec-4d59-8776-95c7c986bb92-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:28:42.506205 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.505723    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83a40616-0eec-4d59-8776-95c7c986bb92-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:28:42.506205 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.505732    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx6vl\" (UniqueName: \"kubernetes.io/projected/83a40616-0eec-4d59-8776-95c7c986bb92-kube-api-access-gx6vl\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:28:42.871744 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.871658    2574 generic.go:356] "Generic (PLEG): container finished" podID="83a40616-0eec-4d59-8776-95c7c986bb92" containerID="a3623bd5791f2d290908c7e0effd4382dca58fe3eaff0da20e54601547c8b792" exitCode=0
Aug 05 17:28:42.871744 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.871721    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"
Aug 05 17:28:42.871923 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.871749    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" event={"ID":"83a40616-0eec-4d59-8776-95c7c986bb92","Type":"ContainerDied","Data":"a3623bd5791f2d290908c7e0effd4382dca58fe3eaff0da20e54601547c8b792"}
Aug 05 17:28:42.871923 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.871814    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w" event={"ID":"83a40616-0eec-4d59-8776-95c7c986bb92","Type":"ContainerDied","Data":"cb5edd1ded0f6ddea0cec929913e6541ee670a9846321d27eb1efb25439f5477"}
Aug 05 17:28:42.871923 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.871842    2574 scope.go:117] "RemoveContainer" containerID="4a8cfb89cd93a3d09c390072a51b3707b675acb602072ba21537ce187d797bd0"
Aug 05 17:28:42.880188 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.880169    2574 scope.go:117] "RemoveContainer" containerID="a3623bd5791f2d290908c7e0effd4382dca58fe3eaff0da20e54601547c8b792"
Aug 05 17:28:42.887185 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.887162    2574 scope.go:117] "RemoveContainer" containerID="3f2b63b7e0209b56447d39c09a97165e5717a2f07bfa8015f1e2f9b075c49ca8"
Aug 05 17:28:42.894209 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.894182    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"]
Aug 05 17:28:42.894930 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.894914    2574 scope.go:117] "RemoveContainer" containerID="4a8cfb89cd93a3d09c390072a51b3707b675acb602072ba21537ce187d797bd0"
Aug 05 17:28:42.895220 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:28:42.895202    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8cfb89cd93a3d09c390072a51b3707b675acb602072ba21537ce187d797bd0\": container with ID starting with 4a8cfb89cd93a3d09c390072a51b3707b675acb602072ba21537ce187d797bd0 not found: ID does not exist" containerID="4a8cfb89cd93a3d09c390072a51b3707b675acb602072ba21537ce187d797bd0"
Aug 05 17:28:42.895270 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.895230    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8cfb89cd93a3d09c390072a51b3707b675acb602072ba21537ce187d797bd0"} err="failed to get container status \"4a8cfb89cd93a3d09c390072a51b3707b675acb602072ba21537ce187d797bd0\": rpc error: code = NotFound desc = could not find container \"4a8cfb89cd93a3d09c390072a51b3707b675acb602072ba21537ce187d797bd0\": container with ID starting with 4a8cfb89cd93a3d09c390072a51b3707b675acb602072ba21537ce187d797bd0 not found: ID does not exist"
Aug 05 17:28:42.895270 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.895250    2574 scope.go:117] "RemoveContainer" containerID="a3623bd5791f2d290908c7e0effd4382dca58fe3eaff0da20e54601547c8b792"
Aug 05 17:28:42.895489 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:28:42.895472    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3623bd5791f2d290908c7e0effd4382dca58fe3eaff0da20e54601547c8b792\": container with ID starting with a3623bd5791f2d290908c7e0effd4382dca58fe3eaff0da20e54601547c8b792 not found: ID does not exist" containerID="a3623bd5791f2d290908c7e0effd4382dca58fe3eaff0da20e54601547c8b792"
Aug 05 17:28:42.895529 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.895498    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3623bd5791f2d290908c7e0effd4382dca58fe3eaff0da20e54601547c8b792"} err="failed to get container status \"a3623bd5791f2d290908c7e0effd4382dca58fe3eaff0da20e54601547c8b792\": rpc error: code = NotFound desc = could not find container \"a3623bd5791f2d290908c7e0effd4382dca58fe3eaff0da20e54601547c8b792\": container with ID starting with a3623bd5791f2d290908c7e0effd4382dca58fe3eaff0da20e54601547c8b792 not found: ID does not exist"
Aug 05 17:28:42.895529 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.895514    2574 scope.go:117] "RemoveContainer" containerID="3f2b63b7e0209b56447d39c09a97165e5717a2f07bfa8015f1e2f9b075c49ca8"
Aug 05 17:28:42.895734 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:28:42.895718    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2b63b7e0209b56447d39c09a97165e5717a2f07bfa8015f1e2f9b075c49ca8\": container with ID starting with 3f2b63b7e0209b56447d39c09a97165e5717a2f07bfa8015f1e2f9b075c49ca8 not found: ID does not exist" containerID="3f2b63b7e0209b56447d39c09a97165e5717a2f07bfa8015f1e2f9b075c49ca8"
Aug 05 17:28:42.895774 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.895737    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2b63b7e0209b56447d39c09a97165e5717a2f07bfa8015f1e2f9b075c49ca8"} err="failed to get container status \"3f2b63b7e0209b56447d39c09a97165e5717a2f07bfa8015f1e2f9b075c49ca8\": rpc error: code = NotFound desc = could not find container \"3f2b63b7e0209b56447d39c09a97165e5717a2f07bfa8015f1e2f9b075c49ca8\": container with ID starting with 3f2b63b7e0209b56447d39c09a97165e5717a2f07bfa8015f1e2f9b075c49ca8 not found: ID does not exist"
Aug 05 17:28:42.898993 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:42.898968    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-849e6-predictor-65d8f5449-qq64w"]
Aug 05 17:28:43.556308 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.556281    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:28:43.613459 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.613342    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kghvj\" (UniqueName: \"kubernetes.io/projected/e152081d-0127-44e1-907d-751da8caab98-kube-api-access-kghvj\") pod \"e152081d-0127-44e1-907d-751da8caab98\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") "
Aug 05 17:28:43.613459 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.613413    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e152081d-0127-44e1-907d-751da8caab98-kube-rbac-proxy-sar-config\") pod \"e152081d-0127-44e1-907d-751da8caab98\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") "
Aug 05 17:28:43.613459 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.613439    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e152081d-0127-44e1-907d-751da8caab98-cabundle-cert\") pod \"e152081d-0127-44e1-907d-751da8caab98\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") "
Aug 05 17:28:43.613737 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.613479    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e152081d-0127-44e1-907d-751da8caab98-kserve-provision-location\") pod \"e152081d-0127-44e1-907d-751da8caab98\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") "
Aug 05 17:28:43.613737 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.613503    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e152081d-0127-44e1-907d-751da8caab98-proxy-tls\") pod \"e152081d-0127-44e1-907d-751da8caab98\" (UID: \"e152081d-0127-44e1-907d-751da8caab98\") "
Aug 05 17:28:43.613808 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.613739    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e152081d-0127-44e1-907d-751da8caab98-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e152081d-0127-44e1-907d-751da8caab98" (UID: "e152081d-0127-44e1-907d-751da8caab98"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:28:43.613839 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.613797    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e152081d-0127-44e1-907d-751da8caab98-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e152081d-0127-44e1-907d-751da8caab98" (UID: "e152081d-0127-44e1-907d-751da8caab98"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:28:43.613871 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.613841    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e152081d-0127-44e1-907d-751da8caab98-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e152081d-0127-44e1-907d-751da8caab98" (UID: "e152081d-0127-44e1-907d-751da8caab98"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:28:43.615607 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.615578    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e152081d-0127-44e1-907d-751da8caab98-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e152081d-0127-44e1-907d-751da8caab98" (UID: "e152081d-0127-44e1-907d-751da8caab98"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:28:43.615740 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.615669    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e152081d-0127-44e1-907d-751da8caab98-kube-api-access-kghvj" (OuterVolumeSpecName: "kube-api-access-kghvj") pod "e152081d-0127-44e1-907d-751da8caab98" (UID: "e152081d-0127-44e1-907d-751da8caab98"). InnerVolumeSpecName "kube-api-access-kghvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:28:43.714405 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.714362    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kghvj\" (UniqueName: \"kubernetes.io/projected/e152081d-0127-44e1-907d-751da8caab98-kube-api-access-kghvj\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:28:43.714405 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.714400    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e152081d-0127-44e1-907d-751da8caab98-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:28:43.714405 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.714414    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e152081d-0127-44e1-907d-751da8caab98-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:28:43.714638 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.714426    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e152081d-0127-44e1-907d-751da8caab98-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:28:43.714638 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.714439    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e152081d-0127-44e1-907d-751da8caab98-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:28:43.876980 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.876892    2574 generic.go:356] "Generic (PLEG): container finished" podID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerID="c78732b5f79779250e54d43a1284e30608aa424f0a640c390c5b918548d78b30" exitCode=0
Aug 05 17:28:43.877180 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.876969    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" event={"ID":"0d23bb97-f7a8-4727-90f1-06f9fd246e9b","Type":"ContainerDied","Data":"c78732b5f79779250e54d43a1284e30608aa424f0a640c390c5b918548d78b30"}
Aug 05 17:28:43.878830 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.878809    2574 generic.go:356] "Generic (PLEG): container finished" podID="e152081d-0127-44e1-907d-751da8caab98" containerID="c88ba8257ddfef7994ce9a5542af4df1a76adedf564693df36b2d721ece3171f" exitCode=0
Aug 05 17:28:43.878930 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.878857    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"
Aug 05 17:28:43.878930 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.878893    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" event={"ID":"e152081d-0127-44e1-907d-751da8caab98","Type":"ContainerDied","Data":"c88ba8257ddfef7994ce9a5542af4df1a76adedf564693df36b2d721ece3171f"}
Aug 05 17:28:43.879026 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.878951    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5" event={"ID":"e152081d-0127-44e1-907d-751da8caab98","Type":"ContainerDied","Data":"fa447063c66b70138fecc6fb9a6436516baf79d64fb8dc9dd871ccf46ce3fda2"}
Aug 05 17:28:43.879026 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.878992    2574 scope.go:117] "RemoveContainer" containerID="e9d4e03b0e421cc8b9e6b58cdf968a4b3faf6234f48ff8fb1abfd69076ae0f1e"
Aug 05 17:28:43.881450 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.881431    2574 generic.go:356] "Generic (PLEG): container finished" podID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerID="655804b7e52effc277b6f658d28e7d6cea0edb0a4a53d9823db4b58c10d32a26" exitCode=0
Aug 05 17:28:43.881547 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.881517    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" event={"ID":"583d6899-27a8-4b78-90d1-8c7c871a3c4c","Type":"ContainerDied","Data":"655804b7e52effc277b6f658d28e7d6cea0edb0a4a53d9823db4b58c10d32a26"}
Aug 05 17:28:43.890748 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.890727    2574 scope.go:117] "RemoveContainer" containerID="c88ba8257ddfef7994ce9a5542af4df1a76adedf564693df36b2d721ece3171f"
Aug 05 17:28:43.904025 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.903979    2574 scope.go:117] "RemoveContainer" containerID="349e51838cb83cd4a1fda837a4ad110470998a3c8a965fc267e2261cdb43b6ab"
Aug 05 17:28:43.913005 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.912959    2574 scope.go:117] "RemoveContainer" containerID="e9d4e03b0e421cc8b9e6b58cdf968a4b3faf6234f48ff8fb1abfd69076ae0f1e"
Aug 05 17:28:43.913342 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:28:43.913321    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d4e03b0e421cc8b9e6b58cdf968a4b3faf6234f48ff8fb1abfd69076ae0f1e\": container with ID starting with e9d4e03b0e421cc8b9e6b58cdf968a4b3faf6234f48ff8fb1abfd69076ae0f1e not found: ID does not exist" containerID="e9d4e03b0e421cc8b9e6b58cdf968a4b3faf6234f48ff8fb1abfd69076ae0f1e"
Aug 05 17:28:43.913443 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.913349    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d4e03b0e421cc8b9e6b58cdf968a4b3faf6234f48ff8fb1abfd69076ae0f1e"} err="failed to get container status \"e9d4e03b0e421cc8b9e6b58cdf968a4b3faf6234f48ff8fb1abfd69076ae0f1e\": rpc error: code = NotFound desc = could not find container \"e9d4e03b0e421cc8b9e6b58cdf968a4b3faf6234f48ff8fb1abfd69076ae0f1e\": container with ID starting with e9d4e03b0e421cc8b9e6b58cdf968a4b3faf6234f48ff8fb1abfd69076ae0f1e not found: ID does not exist"
Aug 05 17:28:43.913443 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.913367    2574 scope.go:117] "RemoveContainer" containerID="c88ba8257ddfef7994ce9a5542af4df1a76adedf564693df36b2d721ece3171f"
Aug 05 17:28:43.913443 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.913418    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"]
Aug 05 17:28:43.913665 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:28:43.913646    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88ba8257ddfef7994ce9a5542af4df1a76adedf564693df36b2d721ece3171f\": container with ID starting with c88ba8257ddfef7994ce9a5542af4df1a76adedf564693df36b2d721ece3171f not found: ID does not exist" containerID="c88ba8257ddfef7994ce9a5542af4df1a76adedf564693df36b2d721ece3171f"
Aug 05 17:28:43.913729 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.913667    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88ba8257ddfef7994ce9a5542af4df1a76adedf564693df36b2d721ece3171f"} err="failed to get container status \"c88ba8257ddfef7994ce9a5542af4df1a76adedf564693df36b2d721ece3171f\": rpc error: code = NotFound desc = could not find container \"c88ba8257ddfef7994ce9a5542af4df1a76adedf564693df36b2d721ece3171f\": container with ID starting with c88ba8257ddfef7994ce9a5542af4df1a76adedf564693df36b2d721ece3171f not found: ID does not exist"
Aug 05 17:28:43.913729 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.913684    2574 scope.go:117] "RemoveContainer" containerID="349e51838cb83cd4a1fda837a4ad110470998a3c8a965fc267e2261cdb43b6ab"
Aug 05 17:28:43.913948 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:28:43.913926    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"349e51838cb83cd4a1fda837a4ad110470998a3c8a965fc267e2261cdb43b6ab\": container with ID starting with 349e51838cb83cd4a1fda837a4ad110470998a3c8a965fc267e2261cdb43b6ab not found: ID does not exist" containerID="349e51838cb83cd4a1fda837a4ad110470998a3c8a965fc267e2261cdb43b6ab"
Aug 05 17:28:43.914025 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.913956    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"349e51838cb83cd4a1fda837a4ad110470998a3c8a965fc267e2261cdb43b6ab"} err="failed to get container status \"349e51838cb83cd4a1fda837a4ad110470998a3c8a965fc267e2261cdb43b6ab\": rpc error: code = NotFound desc = could not find container \"349e51838cb83cd4a1fda837a4ad110470998a3c8a965fc267e2261cdb43b6ab\": container with ID starting with 349e51838cb83cd4a1fda837a4ad110470998a3c8a965fc267e2261cdb43b6ab not found: ID does not exist"
Aug 05 17:28:43.920636 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:43.920607    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-849e6-predictor-86d8d9db6f-gfrt5"]
Aug 05 17:28:44.193977 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:44.193900    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" path="/var/lib/kubelet/pods/83a40616-0eec-4d59-8776-95c7c986bb92/volumes"
Aug 05 17:28:44.194417 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:44.194404    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e152081d-0127-44e1-907d-751da8caab98" path="/var/lib/kubelet/pods/e152081d-0127-44e1-907d-751da8caab98/volumes"
Aug 05 17:28:44.887084 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:44.887040    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" event={"ID":"0d23bb97-f7a8-4727-90f1-06f9fd246e9b","Type":"ContainerStarted","Data":"c6ea8efa7e40a28952432767cdf634b2a01e232d49b1cdf6b3863ce4d5543e82"}
Aug 05 17:28:44.887587 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:44.887114    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" event={"ID":"0d23bb97-f7a8-4727-90f1-06f9fd246e9b","Type":"ContainerStarted","Data":"13baf8ac2519926ae614112106b50f54e770e66a625325006e540cd81be99446"}
Aug 05 17:28:44.887587 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:44.887368    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:44.890008 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:44.889975    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" event={"ID":"583d6899-27a8-4b78-90d1-8c7c871a3c4c","Type":"ContainerStarted","Data":"1ed6c1921455420bc3712f91f59eae8e9386af5539dec02a0aab592017973f39"}
Aug 05 17:28:44.890134 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:44.890032    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" event={"ID":"583d6899-27a8-4b78-90d1-8c7c871a3c4c","Type":"ContainerStarted","Data":"6d35512407ca43c6d9e916405c77cbb7fe669ef27f497cc813f3ee66f6d8dcd0"}
Aug 05 17:28:44.890277 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:44.890258    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:44.910052 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:44.910003    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" podStartSLOduration=3.101305214 podStartE2EDuration="6.909981741s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.808676527s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:28:44.908265957 +0000 UTC m=+975.200120788" watchObservedRunningTime="2026-08-05 17:28:44.909981741 +0000 UTC m=+975.201836572"
Aug 05 17:28:44.928013 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:44.927951    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" podStartSLOduration=3.124226121 podStartE2EDuration="6.927923617s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.803697496s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:28:44.92752061 +0000 UTC m=+975.219375452" watchObservedRunningTime="2026-08-05 17:28:44.927923617 +0000 UTC m=+975.219778450"
Aug 05 17:28:45.893811 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:45.893781    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:45.894327 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:45.894035    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:45.895086 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:45.895049    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Aug 05 17:28:45.895132 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:45.895070    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Aug 05 17:28:46.898236 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:46.898196    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Aug 05 17:28:46.898726 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:46.898269    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Aug 05 17:28:51.903007 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:51.902920    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:28:51.903467 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:51.903216    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:28:51.903554 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:51.903481    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Aug 05 17:28:51.903945 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:28:51.903918    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Aug 05 17:29:01.904386 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:29:01.904348    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Aug 05 17:29:01.904770 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:29:01.904348    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Aug 05 17:29:11.904372 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:29:11.904334    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Aug 05 17:29:11.904860 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:29:11.904335    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Aug 05 17:29:21.904417 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:29:21.904378    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Aug 05 17:29:21.904800 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:29:21.904376    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Aug 05 17:29:31.903905 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:29:31.903859    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Aug 05 17:29:31.903905 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:29:31.903859    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Aug 05 17:29:41.903511 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:29:41.903468    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Aug 05 17:29:41.903929 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:29:41.903904    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Aug 05 17:29:51.903880 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:29:51.903837    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Aug 05 17:29:51.904394 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:29:51.904236    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:29:55.190030 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:29:55.189982    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Aug 05 17:30:05.190228 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:05.190194    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:30:08.196236 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:08.196199    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"]
Aug 05 17:30:08.196632 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:08.196421    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kserve-container" containerID="cri-o://13baf8ac2519926ae614112106b50f54e770e66a625325006e540cd81be99446" gracePeriod=30
Aug 05 17:30:08.196632 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:08.196471    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kube-rbac-proxy" containerID="cri-o://c6ea8efa7e40a28952432767cdf634b2a01e232d49b1cdf6b3863ce4d5543e82" gracePeriod=30
Aug 05 17:30:09.205160 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:09.205109    2574 generic.go:356] "Generic (PLEG): container finished" podID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerID="c6ea8efa7e40a28952432767cdf634b2a01e232d49b1cdf6b3863ce4d5543e82" exitCode=2
Aug 05 17:30:09.205548 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:09.205185    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" event={"ID":"0d23bb97-f7a8-4727-90f1-06f9fd246e9b","Type":"ContainerDied","Data":"c6ea8efa7e40a28952432767cdf634b2a01e232d49b1cdf6b3863ce4d5543e82"}
Aug 05 17:30:11.899064 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:11.899010    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Aug 05 17:30:12.926875 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:12.926851    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:30:13.001775 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.001747    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxlxs\" (UniqueName: \"kubernetes.io/projected/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kube-api-access-nxlxs\") pod \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") "
Aug 05 17:30:13.001961 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.001791    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-proxy-tls\") pod \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") "
Aug 05 17:30:13.001961 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.001821    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-cabundle-cert\") pod \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") "
Aug 05 17:30:13.001961 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.001839    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kube-rbac-proxy-sar-config\") pod \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") "
Aug 05 17:30:13.001961 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.001861    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kserve-provision-location\") pod \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\" (UID: \"0d23bb97-f7a8-4727-90f1-06f9fd246e9b\") "
Aug 05 17:30:13.002226 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.002202    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d23bb97-f7a8-4727-90f1-06f9fd246e9b" (UID: "0d23bb97-f7a8-4727-90f1-06f9fd246e9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:30:13.002282 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.002215    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d23bb97-f7a8-4727-90f1-06f9fd246e9b" (UID: "0d23bb97-f7a8-4727-90f1-06f9fd246e9b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:30:13.002282 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.002243    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d23bb97-f7a8-4727-90f1-06f9fd246e9b" (UID: "0d23bb97-f7a8-4727-90f1-06f9fd246e9b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:30:13.003933 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.003911    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d23bb97-f7a8-4727-90f1-06f9fd246e9b" (UID: "0d23bb97-f7a8-4727-90f1-06f9fd246e9b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:30:13.004026 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.003937    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kube-api-access-nxlxs" (OuterVolumeSpecName: "kube-api-access-nxlxs") pod "0d23bb97-f7a8-4727-90f1-06f9fd246e9b" (UID: "0d23bb97-f7a8-4727-90f1-06f9fd246e9b"). InnerVolumeSpecName "kube-api-access-nxlxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:30:13.103009 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.102978    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:30:13.103009 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.103006    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:30:13.103009 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.103018    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxlxs\" (UniqueName: \"kubernetes.io/projected/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-kube-api-access-nxlxs\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:30:13.103283 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.103028    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:30:13.103283 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.103037    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d23bb97-f7a8-4727-90f1-06f9fd246e9b-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:30:13.220208 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.220175    2574 generic.go:356] "Generic (PLEG): container finished" podID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerID="13baf8ac2519926ae614112106b50f54e770e66a625325006e540cd81be99446" exitCode=0
Aug 05 17:30:13.220391 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.220230    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"
Aug 05 17:30:13.220391 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.220260    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" event={"ID":"0d23bb97-f7a8-4727-90f1-06f9fd246e9b","Type":"ContainerDied","Data":"13baf8ac2519926ae614112106b50f54e770e66a625325006e540cd81be99446"}
Aug 05 17:30:13.220391 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.220333    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf" event={"ID":"0d23bb97-f7a8-4727-90f1-06f9fd246e9b","Type":"ContainerDied","Data":"146b7086e056687a060b0c252f8e663790273b47b33b2338bd798fcfb81f6de0"}
Aug 05 17:30:13.220391 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.220370    2574 scope.go:117] "RemoveContainer" containerID="c6ea8efa7e40a28952432767cdf634b2a01e232d49b1cdf6b3863ce4d5543e82"
Aug 05 17:30:13.229129 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.229111    2574 scope.go:117] "RemoveContainer" containerID="13baf8ac2519926ae614112106b50f54e770e66a625325006e540cd81be99446"
Aug 05 17:30:13.237317 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.237298    2574 scope.go:117] "RemoveContainer" containerID="c78732b5f79779250e54d43a1284e30608aa424f0a640c390c5b918548d78b30"
Aug 05 17:30:13.243653 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.243630    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"]
Aug 05 17:30:13.245019 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.245005    2574 scope.go:117] "RemoveContainer" containerID="c6ea8efa7e40a28952432767cdf634b2a01e232d49b1cdf6b3863ce4d5543e82"
Aug 05 17:30:13.245313 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:30:13.245296    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ea8efa7e40a28952432767cdf634b2a01e232d49b1cdf6b3863ce4d5543e82\": container with ID starting with c6ea8efa7e40a28952432767cdf634b2a01e232d49b1cdf6b3863ce4d5543e82 not found: ID does not exist" containerID="c6ea8efa7e40a28952432767cdf634b2a01e232d49b1cdf6b3863ce4d5543e82"
Aug 05 17:30:13.245360 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.245321    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ea8efa7e40a28952432767cdf634b2a01e232d49b1cdf6b3863ce4d5543e82"} err="failed to get container status \"c6ea8efa7e40a28952432767cdf634b2a01e232d49b1cdf6b3863ce4d5543e82\": rpc error: code = NotFound desc = could not find container \"c6ea8efa7e40a28952432767cdf634b2a01e232d49b1cdf6b3863ce4d5543e82\": container with ID starting with c6ea8efa7e40a28952432767cdf634b2a01e232d49b1cdf6b3863ce4d5543e82 not found: ID does not exist"
Aug 05 17:30:13.245360 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.245338    2574 scope.go:117] "RemoveContainer" containerID="13baf8ac2519926ae614112106b50f54e770e66a625325006e540cd81be99446"
Aug 05 17:30:13.245579 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:30:13.245562    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13baf8ac2519926ae614112106b50f54e770e66a625325006e540cd81be99446\": container with ID starting with 13baf8ac2519926ae614112106b50f54e770e66a625325006e540cd81be99446 not found: ID does not exist" containerID="13baf8ac2519926ae614112106b50f54e770e66a625325006e540cd81be99446"
Aug 05 17:30:13.245633 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.245584    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13baf8ac2519926ae614112106b50f54e770e66a625325006e540cd81be99446"} err="failed to get container status \"13baf8ac2519926ae614112106b50f54e770e66a625325006e540cd81be99446\": rpc error: code = NotFound desc = could not find container \"13baf8ac2519926ae614112106b50f54e770e66a625325006e540cd81be99446\": container with ID starting with 13baf8ac2519926ae614112106b50f54e770e66a625325006e540cd81be99446 not found: ID does not exist"
Aug 05 17:30:13.245633 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.245596    2574 scope.go:117] "RemoveContainer" containerID="c78732b5f79779250e54d43a1284e30608aa424f0a640c390c5b918548d78b30"
Aug 05 17:30:13.245822 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:30:13.245806    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c78732b5f79779250e54d43a1284e30608aa424f0a640c390c5b918548d78b30\": container with ID starting with c78732b5f79779250e54d43a1284e30608aa424f0a640c390c5b918548d78b30 not found: ID does not exist" containerID="c78732b5f79779250e54d43a1284e30608aa424f0a640c390c5b918548d78b30"
Aug 05 17:30:13.245863 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.245825    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78732b5f79779250e54d43a1284e30608aa424f0a640c390c5b918548d78b30"} err="failed to get container status \"c78732b5f79779250e54d43a1284e30608aa424f0a640c390c5b918548d78b30\": rpc error: code = NotFound desc = could not find container \"c78732b5f79779250e54d43a1284e30608aa424f0a640c390c5b918548d78b30\": container with ID starting with c78732b5f79779250e54d43a1284e30608aa424f0a640c390c5b918548d78b30 not found: ID does not exist"
Aug 05 17:30:13.252207 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:13.252158    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-779884cd8f-zpgkf"]
Aug 05 17:30:14.193957 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:14.193925    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" path="/var/lib/kubelet/pods/0d23bb97-f7a8-4727-90f1-06f9fd246e9b/volumes"
Aug 05 17:30:18.535112 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535062    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"]
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535356    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kube-rbac-proxy"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535368    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kube-rbac-proxy"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535379    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="storage-initializer"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535386    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="storage-initializer"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535393    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kube-rbac-proxy"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535399    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kube-rbac-proxy"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535406    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kserve-container"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535411    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kserve-container"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535418    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kserve-container"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535424    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kserve-container"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535431    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="storage-initializer"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535436    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="storage-initializer"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535444    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kube-rbac-proxy"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535449    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kube-rbac-proxy"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535457    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="storage-initializer"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535465    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="storage-initializer"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535478    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kserve-container"
Aug 05 17:30:18.535531 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535486    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kserve-container"
Aug 05 17:30:18.536152 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535547    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kube-rbac-proxy"
Aug 05 17:30:18.536152 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535558    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kube-rbac-proxy"
Aug 05 17:30:18.536152 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535567    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kube-rbac-proxy"
Aug 05 17:30:18.536152 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535575    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d23bb97-f7a8-4727-90f1-06f9fd246e9b" containerName="kserve-container"
Aug 05 17:30:18.536152 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535586    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a40616-0eec-4d59-8776-95c7c986bb92" containerName="kserve-container"
Aug 05 17:30:18.536152 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.535601    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e152081d-0127-44e1-907d-751da8caab98" containerName="kserve-container"
Aug 05 17:30:18.539808 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.539790    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:18.546270 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.546249    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d2f65d-kube-rbac-proxy-sar-config\""
Aug 05 17:30:18.546332 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.546295    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d2f65d-predictor-serving-cert\""
Aug 05 17:30:18.554488 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.554462    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"]
Aug 05 17:30:18.642918 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.642877    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ad5053-e727-45e6-b637-dac908e262fa-kserve-provision-location\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:18.642918 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.642921    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ad5053-e727-45e6-b637-dac908e262fa-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:18.643142 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.642976    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ad5053-e727-45e6-b637-dac908e262fa-proxy-tls\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:18.643142 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.643017    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgfsc\" (UniqueName: \"kubernetes.io/projected/f5ad5053-e727-45e6-b637-dac908e262fa-kube-api-access-dgfsc\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:18.643142 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.643039    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ad5053-e727-45e6-b637-dac908e262fa-cabundle-cert\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:18.744233 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.744194    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ad5053-e727-45e6-b637-dac908e262fa-kserve-provision-location\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:18.744233 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.744238    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ad5053-e727-45e6-b637-dac908e262fa-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:18.744427 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.744269    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ad5053-e727-45e6-b637-dac908e262fa-proxy-tls\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:18.744427 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.744305    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgfsc\" (UniqueName: \"kubernetes.io/projected/f5ad5053-e727-45e6-b637-dac908e262fa-kube-api-access-dgfsc\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:18.744427 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.744324    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ad5053-e727-45e6-b637-dac908e262fa-cabundle-cert\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:18.744574 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:30:18.744434    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-serving-cert: secret "isvc-primary-d2f65d-predictor-serving-cert" not found
Aug 05 17:30:18.744574 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:30:18.744538    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5ad5053-e727-45e6-b637-dac908e262fa-proxy-tls podName:f5ad5053-e727-45e6-b637-dac908e262fa nodeName:}" failed. No retries permitted until 2026-08-05 17:30:19.244517124 +0000 UTC m=+1069.536371936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f5ad5053-e727-45e6-b637-dac908e262fa-proxy-tls") pod "isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" (UID: "f5ad5053-e727-45e6-b637-dac908e262fa") : secret "isvc-primary-d2f65d-predictor-serving-cert" not found
Aug 05 17:30:18.744690 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.744596    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ad5053-e727-45e6-b637-dac908e262fa-kserve-provision-location\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:18.744936 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.744915    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ad5053-e727-45e6-b637-dac908e262fa-cabundle-cert\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:18.745007 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.744948    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ad5053-e727-45e6-b637-dac908e262fa-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:18.753378 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:18.753356    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgfsc\" (UniqueName: \"kubernetes.io/projected/f5ad5053-e727-45e6-b637-dac908e262fa-kube-api-access-dgfsc\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:19.249529 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:19.249495    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ad5053-e727-45e6-b637-dac908e262fa-proxy-tls\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:19.251774 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:19.251747    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ad5053-e727-45e6-b637-dac908e262fa-proxy-tls\") pod \"isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") " pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:19.449968 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:19.449932    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:19.580686 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:19.580656    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"]
Aug 05 17:30:19.583145 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:30:19.583120    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ad5053_e727_45e6_b637_dac908e262fa.slice/crio-85b01b2b2e1b9749abc4da1e47b09133c2c57cc05bbf19e3b7846f742bce5385 WatchSource:0}: Error finding container 85b01b2b2e1b9749abc4da1e47b09133c2c57cc05bbf19e3b7846f742bce5385: Status 404 returned error can't find the container with id 85b01b2b2e1b9749abc4da1e47b09133c2c57cc05bbf19e3b7846f742bce5385
Aug 05 17:30:20.250192 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:20.250153    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" event={"ID":"f5ad5053-e727-45e6-b637-dac908e262fa","Type":"ContainerStarted","Data":"e91ac3873d1d6cc618d1c1d43b4f50d071b30f2bee06458ddf2fe57dac3a6948"}
Aug 05 17:30:20.250366 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:20.250211    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" event={"ID":"f5ad5053-e727-45e6-b637-dac908e262fa","Type":"ContainerStarted","Data":"85b01b2b2e1b9749abc4da1e47b09133c2c57cc05bbf19e3b7846f742bce5385"}
Aug 05 17:30:24.265457 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:24.265422    2574 generic.go:356] "Generic (PLEG): container finished" podID="f5ad5053-e727-45e6-b637-dac908e262fa" containerID="e91ac3873d1d6cc618d1c1d43b4f50d071b30f2bee06458ddf2fe57dac3a6948" exitCode=0
Aug 05 17:30:24.265875 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:24.265497    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" event={"ID":"f5ad5053-e727-45e6-b637-dac908e262fa","Type":"ContainerDied","Data":"e91ac3873d1d6cc618d1c1d43b4f50d071b30f2bee06458ddf2fe57dac3a6948"}
Aug 05 17:30:25.271931 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:25.271899    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" event={"ID":"f5ad5053-e727-45e6-b637-dac908e262fa","Type":"ContainerStarted","Data":"7514a275044b837b22bb0e2bdf71114e2756e3f4cdaf9878669164f547133187"}
Aug 05 17:30:25.272343 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:25.271982    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" event={"ID":"f5ad5053-e727-45e6-b637-dac908e262fa","Type":"ContainerStarted","Data":"9d3b4fc98e09e002a5752fa3dc532160bdc67a2f594da22357b4a95661b826a4"}
Aug 05 17:30:25.272343 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:25.272189    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:25.294236 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:25.294191    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" podStartSLOduration=3.493714987 podStartE2EDuration="7.294177348s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.800462361s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:30:25.292775764 +0000 UTC m=+1075.584630596" watchObservedRunningTime="2026-08-05 17:30:25.294177348 +0000 UTC m=+1075.586032177"
Aug 05 17:30:26.275448 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:26.275418    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:26.276622 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:26.276594    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 05 17:30:27.279260 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:27.279224    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 05 17:30:28.492893 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.492859    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"]
Aug 05 17:30:28.493272 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.493116    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kserve-container" containerID="cri-o://6d35512407ca43c6d9e916405c77cbb7fe669ef27f497cc813f3ee66f6d8dcd0" gracePeriod=30
Aug 05 17:30:28.493272 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.493128    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kube-rbac-proxy" containerID="cri-o://1ed6c1921455420bc3712f91f59eae8e9386af5539dec02a0aab592017973f39" gracePeriod=30
Aug 05 17:30:28.565243 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.565206    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"]
Aug 05 17:30:28.568361 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.568345    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:30:28.570884 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.570866    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-f99f6-predictor-serving-cert\""
Aug 05 17:30:28.571008 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.570866    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-f99f6-kube-rbac-proxy-sar-config\""
Aug 05 17:30:28.581957 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.581926    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"]
Aug 05 17:30:28.618034 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.617995    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb85099-d76f-42ee-a979-79d52c7548e9-proxy-tls\") pod \"message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d\" (UID: \"cdb85099-d76f-42ee-a979-79d52c7548e9\") " pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:30:28.618184 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.618069    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdb85099-d76f-42ee-a979-79d52c7548e9-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d\" (UID: \"cdb85099-d76f-42ee-a979-79d52c7548e9\") " pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:30:28.618184 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.618136    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5jc2\" (UniqueName: \"kubernetes.io/projected/cdb85099-d76f-42ee-a979-79d52c7548e9-kube-api-access-c5jc2\") pod \"message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d\" (UID: \"cdb85099-d76f-42ee-a979-79d52c7548e9\") " pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:30:28.719230 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.719187    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb85099-d76f-42ee-a979-79d52c7548e9-proxy-tls\") pod \"message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d\" (UID: \"cdb85099-d76f-42ee-a979-79d52c7548e9\") " pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:30:28.719427 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.719238    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdb85099-d76f-42ee-a979-79d52c7548e9-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d\" (UID: \"cdb85099-d76f-42ee-a979-79d52c7548e9\") " pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:30:28.719427 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.719278    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5jc2\" (UniqueName: \"kubernetes.io/projected/cdb85099-d76f-42ee-a979-79d52c7548e9-kube-api-access-c5jc2\") pod \"message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d\" (UID: \"cdb85099-d76f-42ee-a979-79d52c7548e9\") " pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:30:28.720028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.719998    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdb85099-d76f-42ee-a979-79d52c7548e9-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d\" (UID: \"cdb85099-d76f-42ee-a979-79d52c7548e9\") " pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:30:28.721781 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.721759    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb85099-d76f-42ee-a979-79d52c7548e9-proxy-tls\") pod \"message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d\" (UID: \"cdb85099-d76f-42ee-a979-79d52c7548e9\") " pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:30:28.728231 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.728208    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5jc2\" (UniqueName: \"kubernetes.io/projected/cdb85099-d76f-42ee-a979-79d52c7548e9-kube-api-access-c5jc2\") pod \"message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d\" (UID: \"cdb85099-d76f-42ee-a979-79d52c7548e9\") " pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:30:28.878942 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:28.878891    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:30:29.011206 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:29.011181    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"]
Aug 05 17:30:29.013618 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:30:29.013594    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb85099_d76f_42ee_a979_79d52c7548e9.slice/crio-3d53f4617f2f338fb7588ef168f1aa446d763b04d5186eb7b4b2bcfa1095d966 WatchSource:0}: Error finding container 3d53f4617f2f338fb7588ef168f1aa446d763b04d5186eb7b4b2bcfa1095d966: Status 404 returned error can't find the container with id 3d53f4617f2f338fb7588ef168f1aa446d763b04d5186eb7b4b2bcfa1095d966
Aug 05 17:30:29.289450 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:29.289421    2574 generic.go:356] "Generic (PLEG): container finished" podID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerID="1ed6c1921455420bc3712f91f59eae8e9386af5539dec02a0aab592017973f39" exitCode=2
Aug 05 17:30:29.289642 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:29.289502    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" event={"ID":"583d6899-27a8-4b78-90d1-8c7c871a3c4c","Type":"ContainerDied","Data":"1ed6c1921455420bc3712f91f59eae8e9386af5539dec02a0aab592017973f39"}
Aug 05 17:30:29.290554 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:29.290535    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d" event={"ID":"cdb85099-d76f-42ee-a979-79d52c7548e9","Type":"ContainerStarted","Data":"3d53f4617f2f338fb7588ef168f1aa446d763b04d5186eb7b4b2bcfa1095d966"}
Aug 05 17:30:30.295008 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:30.294979    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d" event={"ID":"cdb85099-d76f-42ee-a979-79d52c7548e9","Type":"ContainerStarted","Data":"9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36"}
Aug 05 17:30:31.301067 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:31.301036    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d" event={"ID":"cdb85099-d76f-42ee-a979-79d52c7548e9","Type":"ContainerStarted","Data":"528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448"}
Aug 05 17:30:31.301496 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:31.301253    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:30:31.301496 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:31.301393    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:30:31.303102 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:31.303069    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:30:31.320302 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:31.320244    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d" podStartSLOduration=2.147807457 podStartE2EDuration="3.320230731s" totalImagesPullingTime="1.172423274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:30:31.318587234 +0000 UTC m=+1081.610442087" watchObservedRunningTime="2026-08-05 17:30:31.320230731 +0000 UTC m=+1081.612085562"
Aug 05 17:30:31.899265 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:31.899222    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Aug 05 17:30:31.903591 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:31.903566    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Aug 05 17:30:32.284287 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.284261    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:30:32.284800 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.284767    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 05 17:30:32.307926 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.307889    2574 generic.go:356] "Generic (PLEG): container finished" podID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerID="6d35512407ca43c6d9e916405c77cbb7fe669ef27f497cc813f3ee66f6d8dcd0" exitCode=0
Aug 05 17:30:32.308335 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.307923    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" event={"ID":"583d6899-27a8-4b78-90d1-8c7c871a3c4c","Type":"ContainerDied","Data":"6d35512407ca43c6d9e916405c77cbb7fe669ef27f497cc813f3ee66f6d8dcd0"}
Aug 05 17:30:32.341992 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.341967    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:30:32.448426 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.448393    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/583d6899-27a8-4b78-90d1-8c7c871a3c4c-proxy-tls\") pod \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") "
Aug 05 17:30:32.448615 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.448435    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/583d6899-27a8-4b78-90d1-8c7c871a3c4c-cabundle-cert\") pod \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") "
Aug 05 17:30:32.448615 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.448468    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kserve-provision-location\") pod \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") "
Aug 05 17:30:32.448615 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.448495    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kube-rbac-proxy-sar-config\") pod \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") "
Aug 05 17:30:32.448615 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.448564    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjgsb\" (UniqueName: \"kubernetes.io/projected/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kube-api-access-jjgsb\") pod \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\" (UID: \"583d6899-27a8-4b78-90d1-8c7c871a3c4c\") "
Aug 05 17:30:32.448838 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.448766    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583d6899-27a8-4b78-90d1-8c7c871a3c4c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "583d6899-27a8-4b78-90d1-8c7c871a3c4c" (UID: "583d6899-27a8-4b78-90d1-8c7c871a3c4c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:30:32.448838 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.448785    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "583d6899-27a8-4b78-90d1-8c7c871a3c4c" (UID: "583d6899-27a8-4b78-90d1-8c7c871a3c4c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:30:32.448943 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.448903    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "583d6899-27a8-4b78-90d1-8c7c871a3c4c" (UID: "583d6899-27a8-4b78-90d1-8c7c871a3c4c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:30:32.450678 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.450649    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kube-api-access-jjgsb" (OuterVolumeSpecName: "kube-api-access-jjgsb") pod "583d6899-27a8-4b78-90d1-8c7c871a3c4c" (UID: "583d6899-27a8-4b78-90d1-8c7c871a3c4c"). InnerVolumeSpecName "kube-api-access-jjgsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:30:32.450797 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.450742    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583d6899-27a8-4b78-90d1-8c7c871a3c4c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "583d6899-27a8-4b78-90d1-8c7c871a3c4c" (UID: "583d6899-27a8-4b78-90d1-8c7c871a3c4c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:30:32.549238 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.549201    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/583d6899-27a8-4b78-90d1-8c7c871a3c4c-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:30:32.549238 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.549232    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/583d6899-27a8-4b78-90d1-8c7c871a3c4c-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:30:32.549238 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.549242    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:30:32.549473 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.549251    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:30:32.549473 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:32.549260    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjgsb\" (UniqueName: \"kubernetes.io/projected/583d6899-27a8-4b78-90d1-8c7c871a3c4c-kube-api-access-jjgsb\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:30:33.317865 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:33.317830    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"
Aug 05 17:30:33.318360 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:33.317835    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6" event={"ID":"583d6899-27a8-4b78-90d1-8c7c871a3c4c","Type":"ContainerDied","Data":"3e36e23abc557e2af511b55db166eb6e96ae7d09f2eb0123c1b1802641bc1a27"}
Aug 05 17:30:33.318360 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:33.317991    2574 scope.go:117] "RemoveContainer" containerID="1ed6c1921455420bc3712f91f59eae8e9386af5539dec02a0aab592017973f39"
Aug 05 17:30:33.326936 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:33.326919    2574 scope.go:117] "RemoveContainer" containerID="6d35512407ca43c6d9e916405c77cbb7fe669ef27f497cc813f3ee66f6d8dcd0"
Aug 05 17:30:33.334412 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:33.334393    2574 scope.go:117] "RemoveContainer" containerID="655804b7e52effc277b6f658d28e7d6cea0edb0a4a53d9823db4b58c10d32a26"
Aug 05 17:30:33.346093 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:33.346052    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"]
Aug 05 17:30:33.351756 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:33.351734    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-f37da-predictor-c787849cc-729z6"]
Aug 05 17:30:34.193616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:34.193583    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" path="/var/lib/kubelet/pods/583d6899-27a8-4b78-90d1-8c7c871a3c4c/volumes"
Aug 05 17:30:38.323187 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.323159    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:30:38.669236 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.669151    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"]
Aug 05 17:30:38.669524 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.669508    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kube-rbac-proxy"
Aug 05 17:30:38.669609 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.669528    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kube-rbac-proxy"
Aug 05 17:30:38.669609 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.669543    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kserve-container"
Aug 05 17:30:38.669609 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.669554    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kserve-container"
Aug 05 17:30:38.669609 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.669583    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="storage-initializer"
Aug 05 17:30:38.669609 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.669593    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="storage-initializer"
Aug 05 17:30:38.669850 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.669728    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kserve-container"
Aug 05 17:30:38.669850 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.669744    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="583d6899-27a8-4b78-90d1-8c7c871a3c4c" containerName="kube-rbac-proxy"
Aug 05 17:30:38.674398 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.674374    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.676984 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.676960    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-f99f6-predictor-serving-cert\""
Aug 05 17:30:38.677685 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.677656    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-f99f6-kube-rbac-proxy-sar-config\""
Aug 05 17:30:38.685409 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.685386    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"]
Aug 05 17:30:38.793582 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.793543    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31da19a9-cf8c-4732-8230-c72ff5f860e8-cabundle-cert\") pod \"isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.793748 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.793598    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vxjr\" (UniqueName: \"kubernetes.io/projected/31da19a9-cf8c-4732-8230-c72ff5f860e8-kube-api-access-4vxjr\") pod \"isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.793748 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.793623    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31da19a9-cf8c-4732-8230-c72ff5f860e8-proxy-tls\") pod \"isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.793748 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.793713    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31da19a9-cf8c-4732-8230-c72ff5f860e8-kserve-provision-location\") pod \"isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.793879 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.793757    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31da19a9-cf8c-4732-8230-c72ff5f860e8-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.894658 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.894613    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31da19a9-cf8c-4732-8230-c72ff5f860e8-proxy-tls\") pod \"isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.894853 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.894681    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31da19a9-cf8c-4732-8230-c72ff5f860e8-kserve-provision-location\") pod \"isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.894853 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.894717    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31da19a9-cf8c-4732-8230-c72ff5f860e8-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.894853 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.894747    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31da19a9-cf8c-4732-8230-c72ff5f860e8-cabundle-cert\") pod \"isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.894853 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.894793    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vxjr\" (UniqueName: \"kubernetes.io/projected/31da19a9-cf8c-4732-8230-c72ff5f860e8-kube-api-access-4vxjr\") pod \"isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.895135 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.895112    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31da19a9-cf8c-4732-8230-c72ff5f860e8-kserve-provision-location\") pod \"isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.895420 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.895398    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31da19a9-cf8c-4732-8230-c72ff5f860e8-cabundle-cert\") pod \"isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.895507 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.895491    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31da19a9-cf8c-4732-8230-c72ff5f860e8-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.897163 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.897144    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31da19a9-cf8c-4732-8230-c72ff5f860e8-proxy-tls\") pod \"isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.905303 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.905279    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vxjr\" (UniqueName: \"kubernetes.io/projected/31da19a9-cf8c-4732-8230-c72ff5f860e8-kube-api-access-4vxjr\") pod \"isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:38.986459 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:38.986366    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:39.124434 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:39.124396    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"]
Aug 05 17:30:39.128892 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:30:39.128862    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31da19a9_cf8c_4732_8230_c72ff5f860e8.slice/crio-a6e5c393e1a9803d5b60a7fd71576a824be955419da65819c5bc9eaffada4c31 WatchSource:0}: Error finding container a6e5c393e1a9803d5b60a7fd71576a824be955419da65819c5bc9eaffada4c31: Status 404 returned error can't find the container with id a6e5c393e1a9803d5b60a7fd71576a824be955419da65819c5bc9eaffada4c31
Aug 05 17:30:39.342286 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:39.342245    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" event={"ID":"31da19a9-cf8c-4732-8230-c72ff5f860e8","Type":"ContainerStarted","Data":"0e2aabe751754f3c54326fa4c30a397ac41f775382b098ebcb6903a5db99918f"}
Aug 05 17:30:39.342781 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:39.342320    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" event={"ID":"31da19a9-cf8c-4732-8230-c72ff5f860e8","Type":"ContainerStarted","Data":"a6e5c393e1a9803d5b60a7fd71576a824be955419da65819c5bc9eaffada4c31"}
Aug 05 17:30:42.285326 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:42.285287    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 05 17:30:43.357946 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:43.357858    2574 generic.go:356] "Generic (PLEG): container finished" podID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerID="0e2aabe751754f3c54326fa4c30a397ac41f775382b098ebcb6903a5db99918f" exitCode=0
Aug 05 17:30:43.357946 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:43.357930    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" event={"ID":"31da19a9-cf8c-4732-8230-c72ff5f860e8","Type":"ContainerDied","Data":"0e2aabe751754f3c54326fa4c30a397ac41f775382b098ebcb6903a5db99918f"}
Aug 05 17:30:44.364807 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:44.364771    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" event={"ID":"31da19a9-cf8c-4732-8230-c72ff5f860e8","Type":"ContainerStarted","Data":"084721994caaaebb40fbd3a2f24cbff14dcb7c79b0b7b4d5f1e2511b41e92588"}
Aug 05 17:30:44.365236 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:44.364840    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" event={"ID":"31da19a9-cf8c-4732-8230-c72ff5f860e8","Type":"ContainerStarted","Data":"a617f0a4e96cf1748ffac2530c8763efc757d668a428f11f17ed9cb5f8f427e1"}
Aug 05 17:30:44.365236 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:44.364865    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" event={"ID":"31da19a9-cf8c-4732-8230-c72ff5f860e8","Type":"ContainerStarted","Data":"44447e57190039aa91d6b5b452e0cbaf1d6bd2a116aa1edc6cb420d2018b898f"}
Aug 05 17:30:44.365236 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:44.365190    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:44.365377 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:44.365316    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:44.366663 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:44.366635    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 05 17:30:44.390092 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:44.390007    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podStartSLOduration=2.518457394 podStartE2EDuration="6.389990244s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.87153285s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:30:44.38881321 +0000 UTC m=+1094.680668066" watchObservedRunningTime="2026-08-05 17:30:44.389990244 +0000 UTC m=+1094.681845075"
Aug 05 17:30:45.368802 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:45.368770    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:45.369227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:45.368826    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 05 17:30:45.370051 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:45.370017    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:30:46.372705 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:46.372658    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 05 17:30:46.373216 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:46.373192    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:30:51.377732 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:51.377702    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:30:51.378455 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:51.378422    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 05 17:30:51.378717 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:51.378684    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:30:52.284946 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:30:52.284905    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 05 17:31:01.379127 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:01.379067    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 05 17:31:01.379701 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:01.379628    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:31:02.285234 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:02.285197    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 05 17:31:11.379241 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:11.379198    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 05 17:31:11.379803 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:11.379571    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:31:12.285515 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:12.285477    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 05 17:31:21.379123 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:21.379053    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 05 17:31:21.379556 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:21.379495    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:31:22.285501 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:22.285461    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 05 17:31:31.378810 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:31.378761    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 05 17:31:31.379300 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:31.379263    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:31:32.284989 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:32.284939    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 05 17:31:37.189257 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:37.189214    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Aug 05 17:31:41.378658 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:41.378614    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 05 17:31:41.379178 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:41.379116    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:31:47.190227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:47.190191    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:31:51.378435 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:51.378389    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 05 17:31:51.378916 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:51.378864    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:31:52.190017 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:52.189969    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 05 17:31:52.190332 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:31:52.190309    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:32:02.193243 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:02.193165    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:32:02.193604 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:02.193248    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:32:13.654883 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:13.654850    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d_cdb85099-d76f-42ee-a979-79d52c7548e9/kserve-container/0.log"
Aug 05 17:32:13.843096 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:13.843040    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"]
Aug 05 17:32:13.843376 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:13.843340    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container" containerID="cri-o://44447e57190039aa91d6b5b452e0cbaf1d6bd2a116aa1edc6cb420d2018b898f" gracePeriod=30
Aug 05 17:32:13.843473 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:13.843410    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent" containerID="cri-o://084721994caaaebb40fbd3a2f24cbff14dcb7c79b0b7b4d5f1e2511b41e92588" gracePeriod=30
Aug 05 17:32:13.843554 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:13.843414    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kube-rbac-proxy" containerID="cri-o://a617f0a4e96cf1748ffac2530c8763efc757d668a428f11f17ed9cb5f8f427e1" gracePeriod=30
Aug 05 17:32:13.891851 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:13.891810    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"]
Aug 05 17:32:13.892109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:13.892059    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d" podUID="cdb85099-d76f-42ee-a979-79d52c7548e9" containerName="kserve-container" containerID="cri-o://9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36" gracePeriod=30
Aug 05 17:32:13.892211 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:13.892105    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d" podUID="cdb85099-d76f-42ee-a979-79d52c7548e9" containerName="kube-rbac-proxy" containerID="cri-o://528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448" gracePeriod=30
Aug 05 17:32:14.136447 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.136422    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:32:14.244734 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.244694    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdb85099-d76f-42ee-a979-79d52c7548e9-kube-rbac-proxy-sar-config\") pod \"cdb85099-d76f-42ee-a979-79d52c7548e9\" (UID: \"cdb85099-d76f-42ee-a979-79d52c7548e9\") "
Aug 05 17:32:14.244916 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.244749    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5jc2\" (UniqueName: \"kubernetes.io/projected/cdb85099-d76f-42ee-a979-79d52c7548e9-kube-api-access-c5jc2\") pod \"cdb85099-d76f-42ee-a979-79d52c7548e9\" (UID: \"cdb85099-d76f-42ee-a979-79d52c7548e9\") "
Aug 05 17:32:14.244916 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.244773    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb85099-d76f-42ee-a979-79d52c7548e9-proxy-tls\") pod \"cdb85099-d76f-42ee-a979-79d52c7548e9\" (UID: \"cdb85099-d76f-42ee-a979-79d52c7548e9\") "
Aug 05 17:32:14.245157 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.245128    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb85099-d76f-42ee-a979-79d52c7548e9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cdb85099-d76f-42ee-a979-79d52c7548e9" (UID: "cdb85099-d76f-42ee-a979-79d52c7548e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:32:14.246893 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.246870    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb85099-d76f-42ee-a979-79d52c7548e9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cdb85099-d76f-42ee-a979-79d52c7548e9" (UID: "cdb85099-d76f-42ee-a979-79d52c7548e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:32:14.246967 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.246908    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb85099-d76f-42ee-a979-79d52c7548e9-kube-api-access-c5jc2" (OuterVolumeSpecName: "kube-api-access-c5jc2") pod "cdb85099-d76f-42ee-a979-79d52c7548e9" (UID: "cdb85099-d76f-42ee-a979-79d52c7548e9"). InnerVolumeSpecName "kube-api-access-c5jc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:32:14.345386 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.345351    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdb85099-d76f-42ee-a979-79d52c7548e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:14.345386 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.345381    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5jc2\" (UniqueName: \"kubernetes.io/projected/cdb85099-d76f-42ee-a979-79d52c7548e9-kube-api-access-c5jc2\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:14.345386 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.345394    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb85099-d76f-42ee-a979-79d52c7548e9-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:14.706886 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.706791    2574 generic.go:356] "Generic (PLEG): container finished" podID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerID="a617f0a4e96cf1748ffac2530c8763efc757d668a428f11f17ed9cb5f8f427e1" exitCode=2
Aug 05 17:32:14.707315 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.706885    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" event={"ID":"31da19a9-cf8c-4732-8230-c72ff5f860e8","Type":"ContainerDied","Data":"a617f0a4e96cf1748ffac2530c8763efc757d668a428f11f17ed9cb5f8f427e1"}
Aug 05 17:32:14.708614 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.708590    2574 generic.go:356] "Generic (PLEG): container finished" podID="cdb85099-d76f-42ee-a979-79d52c7548e9" containerID="528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448" exitCode=2
Aug 05 17:32:14.708614 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.708611    2574 generic.go:356] "Generic (PLEG): container finished" podID="cdb85099-d76f-42ee-a979-79d52c7548e9" containerID="9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36" exitCode=2
Aug 05 17:32:14.708805 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.708629    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d" event={"ID":"cdb85099-d76f-42ee-a979-79d52c7548e9","Type":"ContainerDied","Data":"528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448"}
Aug 05 17:32:14.708805 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.708650    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"
Aug 05 17:32:14.708805 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.708668    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d" event={"ID":"cdb85099-d76f-42ee-a979-79d52c7548e9","Type":"ContainerDied","Data":"9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36"}
Aug 05 17:32:14.708805 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.708689    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d" event={"ID":"cdb85099-d76f-42ee-a979-79d52c7548e9","Type":"ContainerDied","Data":"3d53f4617f2f338fb7588ef168f1aa446d763b04d5186eb7b4b2bcfa1095d966"}
Aug 05 17:32:14.708805 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.708714    2574 scope.go:117] "RemoveContainer" containerID="528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448"
Aug 05 17:32:14.720091 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.720047    2574 scope.go:117] "RemoveContainer" containerID="9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36"
Aug 05 17:32:14.728143 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.728124    2574 scope.go:117] "RemoveContainer" containerID="528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448"
Aug 05 17:32:14.728384 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:14.728367    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448\": container with ID starting with 528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448 not found: ID does not exist" containerID="528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448"
Aug 05 17:32:14.728432 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.728394    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448"} err="failed to get container status \"528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448\": rpc error: code = NotFound desc = could not find container \"528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448\": container with ID starting with 528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448 not found: ID does not exist"
Aug 05 17:32:14.728432 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.728409    2574 scope.go:117] "RemoveContainer" containerID="9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36"
Aug 05 17:32:14.728666 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:14.728644    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36\": container with ID starting with 9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36 not found: ID does not exist" containerID="9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36"
Aug 05 17:32:14.728707 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.728674    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36"} err="failed to get container status \"9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36\": rpc error: code = NotFound desc = could not find container \"9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36\": container with ID starting with 9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36 not found: ID does not exist"
Aug 05 17:32:14.728707 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.728689    2574 scope.go:117] "RemoveContainer" containerID="528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448"
Aug 05 17:32:14.728951 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.728930    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448"} err="failed to get container status \"528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448\": rpc error: code = NotFound desc = could not find container \"528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448\": container with ID starting with 528125cec10d3c3bfd6c09034b6dd4bea607d9adf4b9616a13170ba4daefc448 not found: ID does not exist"
Aug 05 17:32:14.728997 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.728952    2574 scope.go:117] "RemoveContainer" containerID="9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36"
Aug 05 17:32:14.729198 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.729181    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36"} err="failed to get container status \"9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36\": rpc error: code = NotFound desc = could not find container \"9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36\": container with ID starting with 9af43f95383503a08e2d1c5ec9d29c84e318209c2c8425709185f57b3c32ee36 not found: ID does not exist"
Aug 05 17:32:14.731734 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.731714    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"]
Aug 05 17:32:14.736471 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:14.736449    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f99f6-predictor-7855974cd8-pfh7d"]
Aug 05 17:32:16.194302 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:16.194264    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdb85099-d76f-42ee-a979-79d52c7548e9" path="/var/lib/kubelet/pods/cdb85099-d76f-42ee-a979-79d52c7548e9/volumes"
Aug 05 17:32:16.373509 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:16.373465    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Aug 05 17:32:18.729041 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:18.729001    2574 generic.go:356] "Generic (PLEG): container finished" podID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerID="44447e57190039aa91d6b5b452e0cbaf1d6bd2a116aa1edc6cb420d2018b898f" exitCode=0
Aug 05 17:32:18.729543 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:18.729059    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" event={"ID":"31da19a9-cf8c-4732-8230-c72ff5f860e8","Type":"ContainerDied","Data":"44447e57190039aa91d6b5b452e0cbaf1d6bd2a116aa1edc6cb420d2018b898f"}
Aug 05 17:32:21.373666 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:21.373622    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Aug 05 17:32:21.684853 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:21.684769    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"]
Aug 05 17:32:21.685098 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:21.685054    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kserve-container" containerID="cri-o://9d3b4fc98e09e002a5752fa3dc532160bdc67a2f594da22357b4a95661b826a4" gracePeriod=30
Aug 05 17:32:21.685227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:21.685154    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kube-rbac-proxy" containerID="cri-o://7514a275044b837b22bb0e2bdf71114e2756e3f4cdaf9878669164f547133187" gracePeriod=30
Aug 05 17:32:22.190708 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:22.190669    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 05 17:32:22.191045 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:22.191022    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:32:22.280464 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:22.280417    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Aug 05 17:32:22.745539 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:22.745507    2574 generic.go:356] "Generic (PLEG): container finished" podID="f5ad5053-e727-45e6-b637-dac908e262fa" containerID="7514a275044b837b22bb0e2bdf71114e2756e3f4cdaf9878669164f547133187" exitCode=2
Aug 05 17:32:22.745923 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:22.745560    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" event={"ID":"f5ad5053-e727-45e6-b637-dac908e262fa","Type":"ContainerDied","Data":"7514a275044b837b22bb0e2bdf71114e2756e3f4cdaf9878669164f547133187"}
Aug 05 17:32:26.373280 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.373233    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Aug 05 17:32:26.373653 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.373362    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:32:26.532368 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.532340    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:32:26.644482 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.644384    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ad5053-e727-45e6-b637-dac908e262fa-kube-rbac-proxy-sar-config\") pod \"f5ad5053-e727-45e6-b637-dac908e262fa\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") "
Aug 05 17:32:26.644482 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.644432    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ad5053-e727-45e6-b637-dac908e262fa-proxy-tls\") pod \"f5ad5053-e727-45e6-b637-dac908e262fa\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") "
Aug 05 17:32:26.644482 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.644472    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ad5053-e727-45e6-b637-dac908e262fa-kserve-provision-location\") pod \"f5ad5053-e727-45e6-b637-dac908e262fa\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") "
Aug 05 17:32:26.644772 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.644491    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ad5053-e727-45e6-b637-dac908e262fa-cabundle-cert\") pod \"f5ad5053-e727-45e6-b637-dac908e262fa\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") "
Aug 05 17:32:26.644772 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.644534    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgfsc\" (UniqueName: \"kubernetes.io/projected/f5ad5053-e727-45e6-b637-dac908e262fa-kube-api-access-dgfsc\") pod \"f5ad5053-e727-45e6-b637-dac908e262fa\" (UID: \"f5ad5053-e727-45e6-b637-dac908e262fa\") "
Aug 05 17:32:26.644873 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.644838    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ad5053-e727-45e6-b637-dac908e262fa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5ad5053-e727-45e6-b637-dac908e262fa" (UID: "f5ad5053-e727-45e6-b637-dac908e262fa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:32:26.644959 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.644887    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ad5053-e727-45e6-b637-dac908e262fa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5ad5053-e727-45e6-b637-dac908e262fa" (UID: "f5ad5053-e727-45e6-b637-dac908e262fa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:32:26.644959 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.644912    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ad5053-e727-45e6-b637-dac908e262fa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5ad5053-e727-45e6-b637-dac908e262fa" (UID: "f5ad5053-e727-45e6-b637-dac908e262fa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:32:26.646634 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.646609    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ad5053-e727-45e6-b637-dac908e262fa-kube-api-access-dgfsc" (OuterVolumeSpecName: "kube-api-access-dgfsc") pod "f5ad5053-e727-45e6-b637-dac908e262fa" (UID: "f5ad5053-e727-45e6-b637-dac908e262fa"). InnerVolumeSpecName "kube-api-access-dgfsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:32:26.646742 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.646631    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ad5053-e727-45e6-b637-dac908e262fa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5ad5053-e727-45e6-b637-dac908e262fa" (UID: "f5ad5053-e727-45e6-b637-dac908e262fa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:32:26.746023 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.745986    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgfsc\" (UniqueName: \"kubernetes.io/projected/f5ad5053-e727-45e6-b637-dac908e262fa-kube-api-access-dgfsc\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:26.746023 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.746015    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ad5053-e727-45e6-b637-dac908e262fa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:26.746023 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.746026    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ad5053-e727-45e6-b637-dac908e262fa-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:26.746332 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.746035    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ad5053-e727-45e6-b637-dac908e262fa-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:26.746332 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.746044    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ad5053-e727-45e6-b637-dac908e262fa-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:26.762128 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.762093    2574 generic.go:356] "Generic (PLEG): container finished" podID="f5ad5053-e727-45e6-b637-dac908e262fa" containerID="9d3b4fc98e09e002a5752fa3dc532160bdc67a2f594da22357b4a95661b826a4" exitCode=0
Aug 05 17:32:26.762283 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.762152    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"
Aug 05 17:32:26.762283 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.762163    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" event={"ID":"f5ad5053-e727-45e6-b637-dac908e262fa","Type":"ContainerDied","Data":"9d3b4fc98e09e002a5752fa3dc532160bdc67a2f594da22357b4a95661b826a4"}
Aug 05 17:32:26.762283 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.762226    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff" event={"ID":"f5ad5053-e727-45e6-b637-dac908e262fa","Type":"ContainerDied","Data":"85b01b2b2e1b9749abc4da1e47b09133c2c57cc05bbf19e3b7846f742bce5385"}
Aug 05 17:32:26.762283 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.762251    2574 scope.go:117] "RemoveContainer" containerID="7514a275044b837b22bb0e2bdf71114e2756e3f4cdaf9878669164f547133187"
Aug 05 17:32:26.770546 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.770528    2574 scope.go:117] "RemoveContainer" containerID="9d3b4fc98e09e002a5752fa3dc532160bdc67a2f594da22357b4a95661b826a4"
Aug 05 17:32:26.778115 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.777954    2574 scope.go:117] "RemoveContainer" containerID="e91ac3873d1d6cc618d1c1d43b4f50d071b30f2bee06458ddf2fe57dac3a6948"
Aug 05 17:32:26.787758 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.787723    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"]
Aug 05 17:32:26.787883 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.787770    2574 scope.go:117] "RemoveContainer" containerID="7514a275044b837b22bb0e2bdf71114e2756e3f4cdaf9878669164f547133187"
Aug 05 17:32:26.788167 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:26.788136    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7514a275044b837b22bb0e2bdf71114e2756e3f4cdaf9878669164f547133187\": container with ID starting with 7514a275044b837b22bb0e2bdf71114e2756e3f4cdaf9878669164f547133187 not found: ID does not exist" containerID="7514a275044b837b22bb0e2bdf71114e2756e3f4cdaf9878669164f547133187"
Aug 05 17:32:26.788266 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.788178    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7514a275044b837b22bb0e2bdf71114e2756e3f4cdaf9878669164f547133187"} err="failed to get container status \"7514a275044b837b22bb0e2bdf71114e2756e3f4cdaf9878669164f547133187\": rpc error: code = NotFound desc = could not find container \"7514a275044b837b22bb0e2bdf71114e2756e3f4cdaf9878669164f547133187\": container with ID starting with 7514a275044b837b22bb0e2bdf71114e2756e3f4cdaf9878669164f547133187 not found: ID does not exist"
Aug 05 17:32:26.788266 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.788204    2574 scope.go:117] "RemoveContainer" containerID="9d3b4fc98e09e002a5752fa3dc532160bdc67a2f594da22357b4a95661b826a4"
Aug 05 17:32:26.788528 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:26.788506    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3b4fc98e09e002a5752fa3dc532160bdc67a2f594da22357b4a95661b826a4\": container with ID starting with 9d3b4fc98e09e002a5752fa3dc532160bdc67a2f594da22357b4a95661b826a4 not found: ID does not exist" containerID="9d3b4fc98e09e002a5752fa3dc532160bdc67a2f594da22357b4a95661b826a4"
Aug 05 17:32:26.788574 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.788537    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3b4fc98e09e002a5752fa3dc532160bdc67a2f594da22357b4a95661b826a4"} err="failed to get container status \"9d3b4fc98e09e002a5752fa3dc532160bdc67a2f594da22357b4a95661b826a4\": rpc error: code = NotFound desc = could not find container \"9d3b4fc98e09e002a5752fa3dc532160bdc67a2f594da22357b4a95661b826a4\": container with ID starting with 9d3b4fc98e09e002a5752fa3dc532160bdc67a2f594da22357b4a95661b826a4 not found: ID does not exist"
Aug 05 17:32:26.788574 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.788560    2574 scope.go:117] "RemoveContainer" containerID="e91ac3873d1d6cc618d1c1d43b4f50d071b30f2bee06458ddf2fe57dac3a6948"
Aug 05 17:32:26.788825 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:26.788804    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91ac3873d1d6cc618d1c1d43b4f50d071b30f2bee06458ddf2fe57dac3a6948\": container with ID starting with e91ac3873d1d6cc618d1c1d43b4f50d071b30f2bee06458ddf2fe57dac3a6948 not found: ID does not exist" containerID="e91ac3873d1d6cc618d1c1d43b4f50d071b30f2bee06458ddf2fe57dac3a6948"
Aug 05 17:32:26.788905 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.788833    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91ac3873d1d6cc618d1c1d43b4f50d071b30f2bee06458ddf2fe57dac3a6948"} err="failed to get container status \"e91ac3873d1d6cc618d1c1d43b4f50d071b30f2bee06458ddf2fe57dac3a6948\": rpc error: code = NotFound desc = could not find container \"e91ac3873d1d6cc618d1c1d43b4f50d071b30f2bee06458ddf2fe57dac3a6948\": container with ID starting with e91ac3873d1d6cc618d1c1d43b4f50d071b30f2bee06458ddf2fe57dac3a6948 not found: ID does not exist"
Aug 05 17:32:26.794841 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.794818    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d2f65d-predictor-74b464cbd4-fs9ff"]
Aug 05 17:32:26.859240 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.859205    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"]
Aug 05 17:32:26.859473 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.859462    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kserve-container"
Aug 05 17:32:26.859528 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.859474    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kserve-container"
Aug 05 17:32:26.859528 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.859487    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="storage-initializer"
Aug 05 17:32:26.859528 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.859493    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="storage-initializer"
Aug 05 17:32:26.859528 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.859499    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb85099-d76f-42ee-a979-79d52c7548e9" containerName="kube-rbac-proxy"
Aug 05 17:32:26.859528 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.859505    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb85099-d76f-42ee-a979-79d52c7548e9" containerName="kube-rbac-proxy"
Aug 05 17:32:26.859528 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.859515    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kube-rbac-proxy"
Aug 05 17:32:26.859528 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.859521    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kube-rbac-proxy"
Aug 05 17:32:26.859738 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.859531    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb85099-d76f-42ee-a979-79d52c7548e9" containerName="kserve-container"
Aug 05 17:32:26.859738 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.859538    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb85099-d76f-42ee-a979-79d52c7548e9" containerName="kserve-container"
Aug 05 17:32:26.859738 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.859578    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kserve-container"
Aug 05 17:32:26.859738 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.859586    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb85099-d76f-42ee-a979-79d52c7548e9" containerName="kube-rbac-proxy"
Aug 05 17:32:26.859738 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.859593    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb85099-d76f-42ee-a979-79d52c7548e9" containerName="kserve-container"
Aug 05 17:32:26.859738 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.859600    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" containerName="kube-rbac-proxy"
Aug 05 17:32:26.863981 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.863948    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:26.866351 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.866332    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-96964e-dockercfg-xzh6h\""
Aug 05 17:32:26.866477 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.866335    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-96964e\""
Aug 05 17:32:26.866577 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.866562    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-96964e-predictor-serving-cert\""
Aug 05 17:32:26.866622 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.866597    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-96964e-kube-rbac-proxy-sar-config\""
Aug 05 17:32:26.874401 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.874380    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"]
Aug 05 17:32:26.947654 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.947554    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec30c76e-92c8-44bf-903d-4ce98e12a203-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:26.947654 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.947595    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6fs5\" (UniqueName: \"kubernetes.io/projected/ec30c76e-92c8-44bf-903d-4ce98e12a203-kube-api-access-k6fs5\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:26.947889 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.947663    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec30c76e-92c8-44bf-903d-4ce98e12a203-cabundle-cert\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:26.947889 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.947705    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec30c76e-92c8-44bf-903d-4ce98e12a203-kserve-provision-location\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:26.947889 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:26.947807    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:27.049136 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:27.049099    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:27.049136 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:27.049144    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec30c76e-92c8-44bf-903d-4ce98e12a203-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:27.049407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:27.049162    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6fs5\" (UniqueName: \"kubernetes.io/projected/ec30c76e-92c8-44bf-903d-4ce98e12a203-kube-api-access-k6fs5\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:27.049407 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:27.049280    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-serving-cert: secret "isvc-init-fail-96964e-predictor-serving-cert" not found
Aug 05 17:32:27.049407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:27.049293    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec30c76e-92c8-44bf-903d-4ce98e12a203-cabundle-cert\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:27.049407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:27.049338    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec30c76e-92c8-44bf-903d-4ce98e12a203-kserve-provision-location\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:27.049407 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:27.049363    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls podName:ec30c76e-92c8-44bf-903d-4ce98e12a203 nodeName:}" failed. No retries permitted until 2026-08-05 17:32:27.549342265 +0000 UTC m=+1197.841197081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls") pod "isvc-init-fail-96964e-predictor-56bf8fc499-zw626" (UID: "ec30c76e-92c8-44bf-903d-4ce98e12a203") : secret "isvc-init-fail-96964e-predictor-serving-cert" not found
Aug 05 17:32:27.049637 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:27.049621    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec30c76e-92c8-44bf-903d-4ce98e12a203-kserve-provision-location\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:27.049830 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:27.049813    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec30c76e-92c8-44bf-903d-4ce98e12a203-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:27.049914 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:27.049898    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec30c76e-92c8-44bf-903d-4ce98e12a203-cabundle-cert\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:27.059342 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:27.059313    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6fs5\" (UniqueName: \"kubernetes.io/projected/ec30c76e-92c8-44bf-903d-4ce98e12a203-kube-api-access-k6fs5\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:27.554162 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:27.554129    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:27.556498 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:27.556469    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls\") pod \"isvc-init-fail-96964e-predictor-56bf8fc499-zw626\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") " pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:27.774789 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:27.774749    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:27.907968 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:27.907941    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"]
Aug 05 17:32:27.910260 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:32:27.910227    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec30c76e_92c8_44bf_903d_4ce98e12a203.slice/crio-1ef78f134b560c44846aa02eafb725a23c4df38588f143b8f5eb1a827c9c6d95 WatchSource:0}: Error finding container 1ef78f134b560c44846aa02eafb725a23c4df38588f143b8f5eb1a827c9c6d95: Status 404 returned error can't find the container with id 1ef78f134b560c44846aa02eafb725a23c4df38588f143b8f5eb1a827c9c6d95
Aug 05 17:32:28.193324 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:28.193251    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5ad5053-e727-45e6-b637-dac908e262fa" path="/var/lib/kubelet/pods/f5ad5053-e727-45e6-b637-dac908e262fa/volumes"
Aug 05 17:32:28.772753 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:28.772715    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626" event={"ID":"ec30c76e-92c8-44bf-903d-4ce98e12a203","Type":"ContainerStarted","Data":"fb19bd6ff10e86e12516ebb8c6ef718309103dbf6b01730de321cba61ce02ccd"}
Aug 05 17:32:28.773282 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:28.772787    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626" event={"ID":"ec30c76e-92c8-44bf-903d-4ce98e12a203","Type":"ContainerStarted","Data":"1ef78f134b560c44846aa02eafb725a23c4df38588f143b8f5eb1a827c9c6d95"}
Aug 05 17:32:30.126505 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:30.126424    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-96964e-predictor-56bf8fc499-zw626_ec30c76e-92c8-44bf-903d-4ce98e12a203/storage-initializer/0.log"
Aug 05 17:32:30.128469 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:30.128447    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-96964e-predictor-56bf8fc499-zw626_ec30c76e-92c8-44bf-903d-4ce98e12a203/storage-initializer/0.log"
Aug 05 17:32:30.782151 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:30.782124    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-96964e-predictor-56bf8fc499-zw626_ec30c76e-92c8-44bf-903d-4ce98e12a203/storage-initializer/0.log"
Aug 05 17:32:30.782306 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:30.782164    2574 generic.go:356] "Generic (PLEG): container finished" podID="ec30c76e-92c8-44bf-903d-4ce98e12a203" containerID="fb19bd6ff10e86e12516ebb8c6ef718309103dbf6b01730de321cba61ce02ccd" exitCode=1
Aug 05 17:32:30.782306 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:30.782222    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626" event={"ID":"ec30c76e-92c8-44bf-903d-4ce98e12a203","Type":"ContainerDied","Data":"fb19bd6ff10e86e12516ebb8c6ef718309103dbf6b01730de321cba61ce02ccd"}
Aug 05 17:32:31.373201 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:31.373146    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Aug 05 17:32:31.788011 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:31.787982    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-96964e-predictor-56bf8fc499-zw626_ec30c76e-92c8-44bf-903d-4ce98e12a203/storage-initializer/0.log"
Aug 05 17:32:31.788200 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:31.788110    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626" event={"ID":"ec30c76e-92c8-44bf-903d-4ce98e12a203","Type":"ContainerStarted","Data":"aaedb3daa2ccb3afa619b0e17a35ec63289dfa1975c8323c8b5916288ad778bf"}
Aug 05 17:32:31.788516 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:31.788496    2574 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626" secret="" err="secret \"fail-s3-sa-96964e-dockercfg-xzh6h\" not found"
Aug 05 17:32:31.850505 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:31.850471    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"]
Aug 05 17:32:31.891178 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:31.891136    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-serving-cert: secret "isvc-init-fail-96964e-predictor-serving-cert" not found
Aug 05 17:32:31.891358 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:31.891239    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls podName:ec30c76e-92c8-44bf-903d-4ce98e12a203 nodeName:}" failed. No retries permitted until 2026-08-05 17:32:32.391216076 +0000 UTC m=+1202.683070894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls") pod "isvc-init-fail-96964e-predictor-56bf8fc499-zw626" (UID: "ec30c76e-92c8-44bf-903d-4ce98e12a203") : secret "isvc-init-fail-96964e-predictor-serving-cert" not found
Aug 05 17:32:32.190156 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:32.190031    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 05 17:32:32.190503 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:32.190477    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:32:32.394085 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:32.394042    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-serving-cert: secret "isvc-init-fail-96964e-predictor-serving-cert" not found
Aug 05 17:32:32.394479 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:32.394149    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls podName:ec30c76e-92c8-44bf-903d-4ce98e12a203 nodeName:}" failed. No retries permitted until 2026-08-05 17:32:33.394133378 +0000 UTC m=+1203.685988189 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls") pod "isvc-init-fail-96964e-predictor-56bf8fc499-zw626" (UID: "ec30c76e-92c8-44bf-903d-4ce98e12a203") : secret "isvc-init-fail-96964e-predictor-serving-cert" not found
Aug 05 17:32:32.791913 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:32.791859    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626" podUID="ec30c76e-92c8-44bf-903d-4ce98e12a203" containerName="storage-initializer" containerID="cri-o://aaedb3daa2ccb3afa619b0e17a35ec63289dfa1975c8323c8b5916288ad778bf" gracePeriod=30
Aug 05 17:32:33.401501 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:33.401460    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-serving-cert: secret "isvc-init-fail-96964e-predictor-serving-cert" not found
Aug 05 17:32:33.401893 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:33.401535    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls podName:ec30c76e-92c8-44bf-903d-4ce98e12a203 nodeName:}" failed. No retries permitted until 2026-08-05 17:32:35.401518639 +0000 UTC m=+1205.693373454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls") pod "isvc-init-fail-96964e-predictor-56bf8fc499-zw626" (UID: "ec30c76e-92c8-44bf-903d-4ce98e12a203") : secret "isvc-init-fail-96964e-predictor-serving-cert" not found
Aug 05 17:32:35.418885 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:35.418846    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-serving-cert: secret "isvc-init-fail-96964e-predictor-serving-cert" not found
Aug 05 17:32:35.419389 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:35.418931    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls podName:ec30c76e-92c8-44bf-903d-4ce98e12a203 nodeName:}" failed. No retries permitted until 2026-08-05 17:32:39.418915776 +0000 UTC m=+1209.710770588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls") pod "isvc-init-fail-96964e-predictor-56bf8fc499-zw626" (UID: "ec30c76e-92c8-44bf-903d-4ce98e12a203") : secret "isvc-init-fail-96964e-predictor-serving-cert" not found
Aug 05 17:32:36.338359 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.338336    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-96964e-predictor-56bf8fc499-zw626_ec30c76e-92c8-44bf-903d-4ce98e12a203/storage-initializer/1.log"
Aug 05 17:32:36.338731 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.338718    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-96964e-predictor-56bf8fc499-zw626_ec30c76e-92c8-44bf-903d-4ce98e12a203/storage-initializer/0.log"
Aug 05 17:32:36.340985 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.340969    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:36.373623 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.373527    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Aug 05 17:32:36.527277 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.527231    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec30c76e-92c8-44bf-903d-4ce98e12a203-kube-rbac-proxy-sar-config\") pod \"ec30c76e-92c8-44bf-903d-4ce98e12a203\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") "
Aug 05 17:32:36.527688 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.527298    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec30c76e-92c8-44bf-903d-4ce98e12a203-kserve-provision-location\") pod \"ec30c76e-92c8-44bf-903d-4ce98e12a203\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") "
Aug 05 17:32:36.527688 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.527349    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls\") pod \"ec30c76e-92c8-44bf-903d-4ce98e12a203\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") "
Aug 05 17:32:36.527688 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.527375    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec30c76e-92c8-44bf-903d-4ce98e12a203-cabundle-cert\") pod \"ec30c76e-92c8-44bf-903d-4ce98e12a203\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") "
Aug 05 17:32:36.527688 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.527421    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6fs5\" (UniqueName: \"kubernetes.io/projected/ec30c76e-92c8-44bf-903d-4ce98e12a203-kube-api-access-k6fs5\") pod \"ec30c76e-92c8-44bf-903d-4ce98e12a203\" (UID: \"ec30c76e-92c8-44bf-903d-4ce98e12a203\") "
Aug 05 17:32:36.527688 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.527517    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec30c76e-92c8-44bf-903d-4ce98e12a203-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec30c76e-92c8-44bf-903d-4ce98e12a203" (UID: "ec30c76e-92c8-44bf-903d-4ce98e12a203"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:32:36.527688 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.527641    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec30c76e-92c8-44bf-903d-4ce98e12a203-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:36.527688 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.527668    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec30c76e-92c8-44bf-903d-4ce98e12a203-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec30c76e-92c8-44bf-903d-4ce98e12a203" (UID: "ec30c76e-92c8-44bf-903d-4ce98e12a203"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:32:36.527954 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.527755    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec30c76e-92c8-44bf-903d-4ce98e12a203-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ec30c76e-92c8-44bf-903d-4ce98e12a203" (UID: "ec30c76e-92c8-44bf-903d-4ce98e12a203"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:32:36.529586 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.529563    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec30c76e-92c8-44bf-903d-4ce98e12a203" (UID: "ec30c76e-92c8-44bf-903d-4ce98e12a203"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:32:36.529677 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.529583    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec30c76e-92c8-44bf-903d-4ce98e12a203-kube-api-access-k6fs5" (OuterVolumeSpecName: "kube-api-access-k6fs5") pod "ec30c76e-92c8-44bf-903d-4ce98e12a203" (UID: "ec30c76e-92c8-44bf-903d-4ce98e12a203"). InnerVolumeSpecName "kube-api-access-k6fs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:32:36.629000 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.628905    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6fs5\" (UniqueName: \"kubernetes.io/projected/ec30c76e-92c8-44bf-903d-4ce98e12a203-kube-api-access-k6fs5\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:36.629000 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.628943    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec30c76e-92c8-44bf-903d-4ce98e12a203-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:36.629000 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.628954    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec30c76e-92c8-44bf-903d-4ce98e12a203-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:36.629000 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.628964    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec30c76e-92c8-44bf-903d-4ce98e12a203-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:36.807853 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.807826    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-96964e-predictor-56bf8fc499-zw626_ec30c76e-92c8-44bf-903d-4ce98e12a203/storage-initializer/1.log"
Aug 05 17:32:36.808243 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.808225    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-96964e-predictor-56bf8fc499-zw626_ec30c76e-92c8-44bf-903d-4ce98e12a203/storage-initializer/0.log"
Aug 05 17:32:36.808363 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.808265    2574 generic.go:356] "Generic (PLEG): container finished" podID="ec30c76e-92c8-44bf-903d-4ce98e12a203" containerID="aaedb3daa2ccb3afa619b0e17a35ec63289dfa1975c8323c8b5916288ad778bf" exitCode=1
Aug 05 17:32:36.808363 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.808323    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"
Aug 05 17:32:36.808363 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.808336    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626" event={"ID":"ec30c76e-92c8-44bf-903d-4ce98e12a203","Type":"ContainerDied","Data":"aaedb3daa2ccb3afa619b0e17a35ec63289dfa1975c8323c8b5916288ad778bf"}
Aug 05 17:32:36.808521 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.808404    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626" event={"ID":"ec30c76e-92c8-44bf-903d-4ce98e12a203","Type":"ContainerDied","Data":"1ef78f134b560c44846aa02eafb725a23c4df38588f143b8f5eb1a827c9c6d95"}
Aug 05 17:32:36.808521 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.808444    2574 scope.go:117] "RemoveContainer" containerID="aaedb3daa2ccb3afa619b0e17a35ec63289dfa1975c8323c8b5916288ad778bf"
Aug 05 17:32:36.816677 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.816660    2574 scope.go:117] "RemoveContainer" containerID="fb19bd6ff10e86e12516ebb8c6ef718309103dbf6b01730de321cba61ce02ccd"
Aug 05 17:32:36.823643 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.823624    2574 scope.go:117] "RemoveContainer" containerID="aaedb3daa2ccb3afa619b0e17a35ec63289dfa1975c8323c8b5916288ad778bf"
Aug 05 17:32:36.823886 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:36.823867    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaedb3daa2ccb3afa619b0e17a35ec63289dfa1975c8323c8b5916288ad778bf\": container with ID starting with aaedb3daa2ccb3afa619b0e17a35ec63289dfa1975c8323c8b5916288ad778bf not found: ID does not exist" containerID="aaedb3daa2ccb3afa619b0e17a35ec63289dfa1975c8323c8b5916288ad778bf"
Aug 05 17:32:36.823934 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.823896    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaedb3daa2ccb3afa619b0e17a35ec63289dfa1975c8323c8b5916288ad778bf"} err="failed to get container status \"aaedb3daa2ccb3afa619b0e17a35ec63289dfa1975c8323c8b5916288ad778bf\": rpc error: code = NotFound desc = could not find container \"aaedb3daa2ccb3afa619b0e17a35ec63289dfa1975c8323c8b5916288ad778bf\": container with ID starting with aaedb3daa2ccb3afa619b0e17a35ec63289dfa1975c8323c8b5916288ad778bf not found: ID does not exist"
Aug 05 17:32:36.823934 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.823915    2574 scope.go:117] "RemoveContainer" containerID="fb19bd6ff10e86e12516ebb8c6ef718309103dbf6b01730de321cba61ce02ccd"
Aug 05 17:32:36.826657 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:36.826629    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb19bd6ff10e86e12516ebb8c6ef718309103dbf6b01730de321cba61ce02ccd\": container with ID starting with fb19bd6ff10e86e12516ebb8c6ef718309103dbf6b01730de321cba61ce02ccd not found: ID does not exist" containerID="fb19bd6ff10e86e12516ebb8c6ef718309103dbf6b01730de321cba61ce02ccd"
Aug 05 17:32:36.826747 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.826661    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb19bd6ff10e86e12516ebb8c6ef718309103dbf6b01730de321cba61ce02ccd"} err="failed to get container status \"fb19bd6ff10e86e12516ebb8c6ef718309103dbf6b01730de321cba61ce02ccd\": rpc error: code = NotFound desc = could not find container \"fb19bd6ff10e86e12516ebb8c6ef718309103dbf6b01730de321cba61ce02ccd\": container with ID starting with fb19bd6ff10e86e12516ebb8c6ef718309103dbf6b01730de321cba61ce02ccd not found: ID does not exist"
Aug 05 17:32:36.850614 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.850584    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"]
Aug 05 17:32:36.855288 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:36.855261    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-96964e-predictor-56bf8fc499-zw626"]
Aug 05 17:32:37.077858 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.077824    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"]
Aug 05 17:32:37.078109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.078096    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec30c76e-92c8-44bf-903d-4ce98e12a203" containerName="storage-initializer"
Aug 05 17:32:37.078179 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.078110    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec30c76e-92c8-44bf-903d-4ce98e12a203" containerName="storage-initializer"
Aug 05 17:32:37.078179 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.078166    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec30c76e-92c8-44bf-903d-4ce98e12a203" containerName="storage-initializer"
Aug 05 17:32:37.078257 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.078215    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec30c76e-92c8-44bf-903d-4ce98e12a203" containerName="storage-initializer"
Aug 05 17:32:37.078257 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.078221    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec30c76e-92c8-44bf-903d-4ce98e12a203" containerName="storage-initializer"
Aug 05 17:32:37.078320 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.078264    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec30c76e-92c8-44bf-903d-4ce98e12a203" containerName="storage-initializer"
Aug 05 17:32:37.082659 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.082640    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.085234 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.085209    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-097fb-predictor-serving-cert\""
Aug 05 17:32:37.085234 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.085224    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-097fb-kube-rbac-proxy-sar-config\""
Aug 05 17:32:37.096036 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.096008    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"]
Aug 05 17:32:37.235149 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.235107    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f976491a-8d91-41e3-94a4-3f599c8d646f-proxy-tls\") pod \"raw-sklearn-097fb-predictor-84cf8468f-x4bcl\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") " pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.235339 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.235201    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f976491a-8d91-41e3-94a4-3f599c8d646f-cabundle-cert\") pod \"raw-sklearn-097fb-predictor-84cf8468f-x4bcl\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") " pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.235339 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.235255    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f976491a-8d91-41e3-94a4-3f599c8d646f-kserve-provision-location\") pod \"raw-sklearn-097fb-predictor-84cf8468f-x4bcl\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") " pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.235339 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.235280    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f976491a-8d91-41e3-94a4-3f599c8d646f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-097fb-predictor-84cf8468f-x4bcl\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") " pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.235339 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.235296    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dtkl\" (UniqueName: \"kubernetes.io/projected/f976491a-8d91-41e3-94a4-3f599c8d646f-kube-api-access-2dtkl\") pod \"raw-sklearn-097fb-predictor-84cf8468f-x4bcl\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") " pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.336346 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.336245    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f976491a-8d91-41e3-94a4-3f599c8d646f-kserve-provision-location\") pod \"raw-sklearn-097fb-predictor-84cf8468f-x4bcl\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") " pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.336526 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.336360    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f976491a-8d91-41e3-94a4-3f599c8d646f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-097fb-predictor-84cf8468f-x4bcl\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") " pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.336526 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.336399    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtkl\" (UniqueName: \"kubernetes.io/projected/f976491a-8d91-41e3-94a4-3f599c8d646f-kube-api-access-2dtkl\") pod \"raw-sklearn-097fb-predictor-84cf8468f-x4bcl\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") " pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.336526 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.336470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f976491a-8d91-41e3-94a4-3f599c8d646f-proxy-tls\") pod \"raw-sklearn-097fb-predictor-84cf8468f-x4bcl\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") " pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.336747 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.336529    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f976491a-8d91-41e3-94a4-3f599c8d646f-cabundle-cert\") pod \"raw-sklearn-097fb-predictor-84cf8468f-x4bcl\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") " pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.336747 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.336673    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f976491a-8d91-41e3-94a4-3f599c8d646f-kserve-provision-location\") pod \"raw-sklearn-097fb-predictor-84cf8468f-x4bcl\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") " pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.337025 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.337005    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f976491a-8d91-41e3-94a4-3f599c8d646f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-097fb-predictor-84cf8468f-x4bcl\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") " pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.337184 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.337166    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f976491a-8d91-41e3-94a4-3f599c8d646f-cabundle-cert\") pod \"raw-sklearn-097fb-predictor-84cf8468f-x4bcl\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") " pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.338862 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.338845    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f976491a-8d91-41e3-94a4-3f599c8d646f-proxy-tls\") pod \"raw-sklearn-097fb-predictor-84cf8468f-x4bcl\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") " pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.345085 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.345054    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dtkl\" (UniqueName: \"kubernetes.io/projected/f976491a-8d91-41e3-94a4-3f599c8d646f-kube-api-access-2dtkl\") pod \"raw-sklearn-097fb-predictor-84cf8468f-x4bcl\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") " pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.392070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.392034    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:37.542224 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.542195    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"]
Aug 05 17:32:37.543959 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:32:37.543932    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf976491a_8d91_41e3_94a4_3f599c8d646f.slice/crio-aa93f4c4d2aea6c58a9725f49506596ec578a94f954d42050b6db440ca94f850 WatchSource:0}: Error finding container aa93f4c4d2aea6c58a9725f49506596ec578a94f954d42050b6db440ca94f850: Status 404 returned error can't find the container with id aa93f4c4d2aea6c58a9725f49506596ec578a94f954d42050b6db440ca94f850
Aug 05 17:32:37.813663 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.813620    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" event={"ID":"f976491a-8d91-41e3-94a4-3f599c8d646f","Type":"ContainerStarted","Data":"231b62a63dbd126ffe0b5375ad4037b216997c348598baab5f35e45fcd4d817a"}
Aug 05 17:32:37.813836 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:37.813729    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" event={"ID":"f976491a-8d91-41e3-94a4-3f599c8d646f","Type":"ContainerStarted","Data":"aa93f4c4d2aea6c58a9725f49506596ec578a94f954d42050b6db440ca94f850"}
Aug 05 17:32:38.193909 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:38.193866    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec30c76e-92c8-44bf-903d-4ce98e12a203" path="/var/lib/kubelet/pods/ec30c76e-92c8-44bf-903d-4ce98e12a203/volumes"
Aug 05 17:32:41.373198 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:41.373157    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Aug 05 17:32:41.830702 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:41.830669    2574 generic.go:356] "Generic (PLEG): container finished" podID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerID="231b62a63dbd126ffe0b5375ad4037b216997c348598baab5f35e45fcd4d817a" exitCode=0
Aug 05 17:32:41.831217 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:41.830747    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" event={"ID":"f976491a-8d91-41e3-94a4-3f599c8d646f","Type":"ContainerDied","Data":"231b62a63dbd126ffe0b5375ad4037b216997c348598baab5f35e45fcd4d817a"}
Aug 05 17:32:42.190185 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:42.190061    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Aug 05 17:32:42.190409 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:42.190386    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:32:42.196112 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:42.194620    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:32:42.196112 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:42.194792    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:32:42.837361 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:42.837328    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" event={"ID":"f976491a-8d91-41e3-94a4-3f599c8d646f","Type":"ContainerStarted","Data":"40898ab585b6db9cd01dc6b678c9b4b35147ae749c7dade55728019f282fc32a"}
Aug 05 17:32:42.837734 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:42.837393    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" event={"ID":"f976491a-8d91-41e3-94a4-3f599c8d646f","Type":"ContainerStarted","Data":"73474f0f558fbcb5521d5ce9f0774420d6f547a2be1c0e2fda80497ca003f0c0"}
Aug 05 17:32:42.837734 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:42.837587    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:42.870011 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:42.869953    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" podStartSLOduration=2.103861681 podStartE2EDuration="5.869931127s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.766069446s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:32:42.869506667 +0000 UTC m=+1213.161361500" watchObservedRunningTime="2026-08-05 17:32:42.869931127 +0000 UTC m=+1213.161785958"
Aug 05 17:32:43.841506 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:43.841477    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:43.842806 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:43.842777    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 05 17:32:44.001431 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.001405    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:32:44.092367 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.092272    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31da19a9-cf8c-4732-8230-c72ff5f860e8-cabundle-cert\") pod \"31da19a9-cf8c-4732-8230-c72ff5f860e8\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") "
Aug 05 17:32:44.092367 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.092315    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31da19a9-cf8c-4732-8230-c72ff5f860e8-kube-rbac-proxy-sar-config\") pod \"31da19a9-cf8c-4732-8230-c72ff5f860e8\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") "
Aug 05 17:32:44.092367 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.092343    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vxjr\" (UniqueName: \"kubernetes.io/projected/31da19a9-cf8c-4732-8230-c72ff5f860e8-kube-api-access-4vxjr\") pod \"31da19a9-cf8c-4732-8230-c72ff5f860e8\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") "
Aug 05 17:32:44.092367 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.092364    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31da19a9-cf8c-4732-8230-c72ff5f860e8-proxy-tls\") pod \"31da19a9-cf8c-4732-8230-c72ff5f860e8\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") "
Aug 05 17:32:44.092714 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.092690    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31da19a9-cf8c-4732-8230-c72ff5f860e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31da19a9-cf8c-4732-8230-c72ff5f860e8" (UID: "31da19a9-cf8c-4732-8230-c72ff5f860e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:32:44.092759 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.092711    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31da19a9-cf8c-4732-8230-c72ff5f860e8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "31da19a9-cf8c-4732-8230-c72ff5f860e8" (UID: "31da19a9-cf8c-4732-8230-c72ff5f860e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:32:44.094499 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.094466    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31da19a9-cf8c-4732-8230-c72ff5f860e8-kube-api-access-4vxjr" (OuterVolumeSpecName: "kube-api-access-4vxjr") pod "31da19a9-cf8c-4732-8230-c72ff5f860e8" (UID: "31da19a9-cf8c-4732-8230-c72ff5f860e8"). InnerVolumeSpecName "kube-api-access-4vxjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:32:44.094604 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.094540    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31da19a9-cf8c-4732-8230-c72ff5f860e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31da19a9-cf8c-4732-8230-c72ff5f860e8" (UID: "31da19a9-cf8c-4732-8230-c72ff5f860e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:32:44.192893 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.192866    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31da19a9-cf8c-4732-8230-c72ff5f860e8-kserve-provision-location\") pod \"31da19a9-cf8c-4732-8230-c72ff5f860e8\" (UID: \"31da19a9-cf8c-4732-8230-c72ff5f860e8\") "
Aug 05 17:32:44.193064 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.192991    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31da19a9-cf8c-4732-8230-c72ff5f860e8-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:44.193064 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.193002    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31da19a9-cf8c-4732-8230-c72ff5f860e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:44.193064 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.193012    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vxjr\" (UniqueName: \"kubernetes.io/projected/31da19a9-cf8c-4732-8230-c72ff5f860e8-kube-api-access-4vxjr\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:44.193064 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.193021    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31da19a9-cf8c-4732-8230-c72ff5f860e8-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:44.193307 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.193242    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31da19a9-cf8c-4732-8230-c72ff5f860e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31da19a9-cf8c-4732-8230-c72ff5f860e8" (UID: "31da19a9-cf8c-4732-8230-c72ff5f860e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:32:44.293550 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.293510    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31da19a9-cf8c-4732-8230-c72ff5f860e8-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:32:44.847139 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.847106    2574 generic.go:356] "Generic (PLEG): container finished" podID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerID="084721994caaaebb40fbd3a2f24cbff14dcb7c79b0b7b4d5f1e2511b41e92588" exitCode=0
Aug 05 17:32:44.847565 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.847174    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" event={"ID":"31da19a9-cf8c-4732-8230-c72ff5f860e8","Type":"ContainerDied","Data":"084721994caaaebb40fbd3a2f24cbff14dcb7c79b0b7b4d5f1e2511b41e92588"}
Aug 05 17:32:44.847565 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.847205    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"
Aug 05 17:32:44.847565 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.847238    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm" event={"ID":"31da19a9-cf8c-4732-8230-c72ff5f860e8","Type":"ContainerDied","Data":"a6e5c393e1a9803d5b60a7fd71576a824be955419da65819c5bc9eaffada4c31"}
Aug 05 17:32:44.847565 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.847274    2574 scope.go:117] "RemoveContainer" containerID="084721994caaaebb40fbd3a2f24cbff14dcb7c79b0b7b4d5f1e2511b41e92588"
Aug 05 17:32:44.848014 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.847988    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 05 17:32:44.856970 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.856946    2574 scope.go:117] "RemoveContainer" containerID="a617f0a4e96cf1748ffac2530c8763efc757d668a428f11f17ed9cb5f8f427e1"
Aug 05 17:32:44.864449 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.864429    2574 scope.go:117] "RemoveContainer" containerID="44447e57190039aa91d6b5b452e0cbaf1d6bd2a116aa1edc6cb420d2018b898f"
Aug 05 17:32:44.868290 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.868268    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"]
Aug 05 17:32:44.872417 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.872396    2574 scope.go:117] "RemoveContainer" containerID="0e2aabe751754f3c54326fa4c30a397ac41f775382b098ebcb6903a5db99918f"
Aug 05 17:32:44.873920 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.873896    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f99f6-predictor-765fd8c79b-frtqm"]
Aug 05 17:32:44.879579 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.879562    2574 scope.go:117] "RemoveContainer" containerID="084721994caaaebb40fbd3a2f24cbff14dcb7c79b0b7b4d5f1e2511b41e92588"
Aug 05 17:32:44.879831 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:44.879813    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084721994caaaebb40fbd3a2f24cbff14dcb7c79b0b7b4d5f1e2511b41e92588\": container with ID starting with 084721994caaaebb40fbd3a2f24cbff14dcb7c79b0b7b4d5f1e2511b41e92588 not found: ID does not exist" containerID="084721994caaaebb40fbd3a2f24cbff14dcb7c79b0b7b4d5f1e2511b41e92588"
Aug 05 17:32:44.879890 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.879837    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084721994caaaebb40fbd3a2f24cbff14dcb7c79b0b7b4d5f1e2511b41e92588"} err="failed to get container status \"084721994caaaebb40fbd3a2f24cbff14dcb7c79b0b7b4d5f1e2511b41e92588\": rpc error: code = NotFound desc = could not find container \"084721994caaaebb40fbd3a2f24cbff14dcb7c79b0b7b4d5f1e2511b41e92588\": container with ID starting with 084721994caaaebb40fbd3a2f24cbff14dcb7c79b0b7b4d5f1e2511b41e92588 not found: ID does not exist"
Aug 05 17:32:44.879890 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.879858    2574 scope.go:117] "RemoveContainer" containerID="a617f0a4e96cf1748ffac2530c8763efc757d668a428f11f17ed9cb5f8f427e1"
Aug 05 17:32:44.880068 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:44.880053    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a617f0a4e96cf1748ffac2530c8763efc757d668a428f11f17ed9cb5f8f427e1\": container with ID starting with a617f0a4e96cf1748ffac2530c8763efc757d668a428f11f17ed9cb5f8f427e1 not found: ID does not exist" containerID="a617f0a4e96cf1748ffac2530c8763efc757d668a428f11f17ed9cb5f8f427e1"
Aug 05 17:32:44.880126 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.880086    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a617f0a4e96cf1748ffac2530c8763efc757d668a428f11f17ed9cb5f8f427e1"} err="failed to get container status \"a617f0a4e96cf1748ffac2530c8763efc757d668a428f11f17ed9cb5f8f427e1\": rpc error: code = NotFound desc = could not find container \"a617f0a4e96cf1748ffac2530c8763efc757d668a428f11f17ed9cb5f8f427e1\": container with ID starting with a617f0a4e96cf1748ffac2530c8763efc757d668a428f11f17ed9cb5f8f427e1 not found: ID does not exist"
Aug 05 17:32:44.880126 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.880101    2574 scope.go:117] "RemoveContainer" containerID="44447e57190039aa91d6b5b452e0cbaf1d6bd2a116aa1edc6cb420d2018b898f"
Aug 05 17:32:44.880290 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:44.880272    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44447e57190039aa91d6b5b452e0cbaf1d6bd2a116aa1edc6cb420d2018b898f\": container with ID starting with 44447e57190039aa91d6b5b452e0cbaf1d6bd2a116aa1edc6cb420d2018b898f not found: ID does not exist" containerID="44447e57190039aa91d6b5b452e0cbaf1d6bd2a116aa1edc6cb420d2018b898f"
Aug 05 17:32:44.880327 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.880293    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44447e57190039aa91d6b5b452e0cbaf1d6bd2a116aa1edc6cb420d2018b898f"} err="failed to get container status \"44447e57190039aa91d6b5b452e0cbaf1d6bd2a116aa1edc6cb420d2018b898f\": rpc error: code = NotFound desc = could not find container \"44447e57190039aa91d6b5b452e0cbaf1d6bd2a116aa1edc6cb420d2018b898f\": container with ID starting with 44447e57190039aa91d6b5b452e0cbaf1d6bd2a116aa1edc6cb420d2018b898f not found: ID does not exist"
Aug 05 17:32:44.880327 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.880304    2574 scope.go:117] "RemoveContainer" containerID="0e2aabe751754f3c54326fa4c30a397ac41f775382b098ebcb6903a5db99918f"
Aug 05 17:32:44.880494 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:32:44.880478    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2aabe751754f3c54326fa4c30a397ac41f775382b098ebcb6903a5db99918f\": container with ID starting with 0e2aabe751754f3c54326fa4c30a397ac41f775382b098ebcb6903a5db99918f not found: ID does not exist" containerID="0e2aabe751754f3c54326fa4c30a397ac41f775382b098ebcb6903a5db99918f"
Aug 05 17:32:44.880534 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:44.880498    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2aabe751754f3c54326fa4c30a397ac41f775382b098ebcb6903a5db99918f"} err="failed to get container status \"0e2aabe751754f3c54326fa4c30a397ac41f775382b098ebcb6903a5db99918f\": rpc error: code = NotFound desc = could not find container \"0e2aabe751754f3c54326fa4c30a397ac41f775382b098ebcb6903a5db99918f\": container with ID starting with 0e2aabe751754f3c54326fa4c30a397ac41f775382b098ebcb6903a5db99918f not found: ID does not exist"
Aug 05 17:32:46.193374 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:46.193344    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" path="/var/lib/kubelet/pods/31da19a9-cf8c-4732-8230-c72ff5f860e8/volumes"
Aug 05 17:32:49.851638 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:49.851609    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:32:49.852226 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:49.852182    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 05 17:32:59.852577 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:32:59.852533    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 05 17:33:09.852143 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:33:09.852099    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 05 17:33:19.852612 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:33:19.852570    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 05 17:33:29.852349 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:33:29.852233    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 05 17:33:39.852660 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:33:39.852621    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 05 17:33:49.852359 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:33:49.852313    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 05 17:33:59.852751 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:33:59.852718    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:34:07.217769 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:07.217733    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"]
Aug 05 17:34:07.218284 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:07.218023    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kserve-container" containerID="cri-o://73474f0f558fbcb5521d5ce9f0774420d6f547a2be1c0e2fda80497ca003f0c0" gracePeriod=30
Aug 05 17:34:07.218284 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:07.218118    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kube-rbac-proxy" containerID="cri-o://40898ab585b6db9cd01dc6b678c9b4b35147ae749c7dade55728019f282fc32a" gracePeriod=30
Aug 05 17:34:08.139512 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:08.139480    2574 generic.go:356] "Generic (PLEG): container finished" podID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerID="40898ab585b6db9cd01dc6b678c9b4b35147ae749c7dade55728019f282fc32a" exitCode=2
Aug 05 17:34:08.139697 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:08.139567    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" event={"ID":"f976491a-8d91-41e3-94a4-3f599c8d646f","Type":"ContainerDied","Data":"40898ab585b6db9cd01dc6b678c9b4b35147ae749c7dade55728019f282fc32a"}
Aug 05 17:34:09.848084 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:09.848042    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Aug 05 17:34:09.852313 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:09.852288    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 05 17:34:12.062889 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.062864    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:34:12.142501 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.142468    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f976491a-8d91-41e3-94a4-3f599c8d646f-cabundle-cert\") pod \"f976491a-8d91-41e3-94a4-3f599c8d646f\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") "
Aug 05 17:34:12.142501 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.142507    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f976491a-8d91-41e3-94a4-3f599c8d646f-kserve-provision-location\") pod \"f976491a-8d91-41e3-94a4-3f599c8d646f\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") "
Aug 05 17:34:12.142714 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.142528    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dtkl\" (UniqueName: \"kubernetes.io/projected/f976491a-8d91-41e3-94a4-3f599c8d646f-kube-api-access-2dtkl\") pod \"f976491a-8d91-41e3-94a4-3f599c8d646f\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") "
Aug 05 17:34:12.142714 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.142552    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f976491a-8d91-41e3-94a4-3f599c8d646f-kube-rbac-proxy-sar-config\") pod \"f976491a-8d91-41e3-94a4-3f599c8d646f\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") "
Aug 05 17:34:12.142714 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.142578    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f976491a-8d91-41e3-94a4-3f599c8d646f-proxy-tls\") pod \"f976491a-8d91-41e3-94a4-3f599c8d646f\" (UID: \"f976491a-8d91-41e3-94a4-3f599c8d646f\") "
Aug 05 17:34:12.142892 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.142864    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f976491a-8d91-41e3-94a4-3f599c8d646f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f976491a-8d91-41e3-94a4-3f599c8d646f" (UID: "f976491a-8d91-41e3-94a4-3f599c8d646f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:34:12.142932 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.142870    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f976491a-8d91-41e3-94a4-3f599c8d646f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f976491a-8d91-41e3-94a4-3f599c8d646f" (UID: "f976491a-8d91-41e3-94a4-3f599c8d646f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:34:12.142932 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.142888    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f976491a-8d91-41e3-94a4-3f599c8d646f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f976491a-8d91-41e3-94a4-3f599c8d646f" (UID: "f976491a-8d91-41e3-94a4-3f599c8d646f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:34:12.144721 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.144695    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f976491a-8d91-41e3-94a4-3f599c8d646f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f976491a-8d91-41e3-94a4-3f599c8d646f" (UID: "f976491a-8d91-41e3-94a4-3f599c8d646f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:34:12.144817 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.144799    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f976491a-8d91-41e3-94a4-3f599c8d646f-kube-api-access-2dtkl" (OuterVolumeSpecName: "kube-api-access-2dtkl") pod "f976491a-8d91-41e3-94a4-3f599c8d646f" (UID: "f976491a-8d91-41e3-94a4-3f599c8d646f"). InnerVolumeSpecName "kube-api-access-2dtkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:34:12.155519 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.155491    2574 generic.go:356] "Generic (PLEG): container finished" podID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerID="73474f0f558fbcb5521d5ce9f0774420d6f547a2be1c0e2fda80497ca003f0c0" exitCode=0
Aug 05 17:34:12.155649 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.155527    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" event={"ID":"f976491a-8d91-41e3-94a4-3f599c8d646f","Type":"ContainerDied","Data":"73474f0f558fbcb5521d5ce9f0774420d6f547a2be1c0e2fda80497ca003f0c0"}
Aug 05 17:34:12.155649 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.155548    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"
Aug 05 17:34:12.155649 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.155600    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl" event={"ID":"f976491a-8d91-41e3-94a4-3f599c8d646f","Type":"ContainerDied","Data":"aa93f4c4d2aea6c58a9725f49506596ec578a94f954d42050b6db440ca94f850"}
Aug 05 17:34:12.155649 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.155625    2574 scope.go:117] "RemoveContainer" containerID="40898ab585b6db9cd01dc6b678c9b4b35147ae749c7dade55728019f282fc32a"
Aug 05 17:34:12.163651 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.163627    2574 scope.go:117] "RemoveContainer" containerID="73474f0f558fbcb5521d5ce9f0774420d6f547a2be1c0e2fda80497ca003f0c0"
Aug 05 17:34:12.170582 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.170566    2574 scope.go:117] "RemoveContainer" containerID="231b62a63dbd126ffe0b5375ad4037b216997c348598baab5f35e45fcd4d817a"
Aug 05 17:34:12.178012 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.177985    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"]
Aug 05 17:34:12.178599 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.178577    2574 scope.go:117] "RemoveContainer" containerID="40898ab585b6db9cd01dc6b678c9b4b35147ae749c7dade55728019f282fc32a"
Aug 05 17:34:12.178908 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:34:12.178879    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40898ab585b6db9cd01dc6b678c9b4b35147ae749c7dade55728019f282fc32a\": container with ID starting with 40898ab585b6db9cd01dc6b678c9b4b35147ae749c7dade55728019f282fc32a not found: ID does not exist" containerID="40898ab585b6db9cd01dc6b678c9b4b35147ae749c7dade55728019f282fc32a"
Aug 05 17:34:12.178965 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.178921    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40898ab585b6db9cd01dc6b678c9b4b35147ae749c7dade55728019f282fc32a"} err="failed to get container status \"40898ab585b6db9cd01dc6b678c9b4b35147ae749c7dade55728019f282fc32a\": rpc error: code = NotFound desc = could not find container \"40898ab585b6db9cd01dc6b678c9b4b35147ae749c7dade55728019f282fc32a\": container with ID starting with 40898ab585b6db9cd01dc6b678c9b4b35147ae749c7dade55728019f282fc32a not found: ID does not exist"
Aug 05 17:34:12.178965 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.178940    2574 scope.go:117] "RemoveContainer" containerID="73474f0f558fbcb5521d5ce9f0774420d6f547a2be1c0e2fda80497ca003f0c0"
Aug 05 17:34:12.179179 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:34:12.179160    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73474f0f558fbcb5521d5ce9f0774420d6f547a2be1c0e2fda80497ca003f0c0\": container with ID starting with 73474f0f558fbcb5521d5ce9f0774420d6f547a2be1c0e2fda80497ca003f0c0 not found: ID does not exist" containerID="73474f0f558fbcb5521d5ce9f0774420d6f547a2be1c0e2fda80497ca003f0c0"
Aug 05 17:34:12.179226 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.179184    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73474f0f558fbcb5521d5ce9f0774420d6f547a2be1c0e2fda80497ca003f0c0"} err="failed to get container status \"73474f0f558fbcb5521d5ce9f0774420d6f547a2be1c0e2fda80497ca003f0c0\": rpc error: code = NotFound desc = could not find container \"73474f0f558fbcb5521d5ce9f0774420d6f547a2be1c0e2fda80497ca003f0c0\": container with ID starting with 73474f0f558fbcb5521d5ce9f0774420d6f547a2be1c0e2fda80497ca003f0c0 not found: ID does not exist"
Aug 05 17:34:12.179226 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.179202    2574 scope.go:117] "RemoveContainer" containerID="231b62a63dbd126ffe0b5375ad4037b216997c348598baab5f35e45fcd4d817a"
Aug 05 17:34:12.179445 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:34:12.179428    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231b62a63dbd126ffe0b5375ad4037b216997c348598baab5f35e45fcd4d817a\": container with ID starting with 231b62a63dbd126ffe0b5375ad4037b216997c348598baab5f35e45fcd4d817a not found: ID does not exist" containerID="231b62a63dbd126ffe0b5375ad4037b216997c348598baab5f35e45fcd4d817a"
Aug 05 17:34:12.179486 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.179449    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231b62a63dbd126ffe0b5375ad4037b216997c348598baab5f35e45fcd4d817a"} err="failed to get container status \"231b62a63dbd126ffe0b5375ad4037b216997c348598baab5f35e45fcd4d817a\": rpc error: code = NotFound desc = could not find container \"231b62a63dbd126ffe0b5375ad4037b216997c348598baab5f35e45fcd4d817a\": container with ID starting with 231b62a63dbd126ffe0b5375ad4037b216997c348598baab5f35e45fcd4d817a not found: ID does not exist"
Aug 05 17:34:12.181897 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.181875    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-097fb-predictor-84cf8468f-x4bcl"]
Aug 05 17:34:12.194425 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.194397    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" path="/var/lib/kubelet/pods/f976491a-8d91-41e3-94a4-3f599c8d646f/volumes"
Aug 05 17:34:12.243116 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.243060    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f976491a-8d91-41e3-94a4-3f599c8d646f-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:34:12.243116 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.243108    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f976491a-8d91-41e3-94a4-3f599c8d646f-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:34:12.243116 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.243121    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dtkl\" (UniqueName: \"kubernetes.io/projected/f976491a-8d91-41e3-94a4-3f599c8d646f-kube-api-access-2dtkl\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:34:12.243369 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.243133    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f976491a-8d91-41e3-94a4-3f599c8d646f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:34:12.243369 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.243142    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f976491a-8d91-41e3-94a4-3f599c8d646f-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:34:12.366383 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366347    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"]
Aug 05 17:34:12.366604 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366593    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent"
Aug 05 17:34:12.366663 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366605    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent"
Aug 05 17:34:12.366663 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366616    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kserve-container"
Aug 05 17:34:12.366663 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366622    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kserve-container"
Aug 05 17:34:12.366663 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366629    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="storage-initializer"
Aug 05 17:34:12.366663 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366635    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="storage-initializer"
Aug 05 17:34:12.366663 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366645    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kube-rbac-proxy"
Aug 05 17:34:12.366663 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366651    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kube-rbac-proxy"
Aug 05 17:34:12.366663 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366659    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container"
Aug 05 17:34:12.366663 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366664    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container"
Aug 05 17:34:12.366931 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366672    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="storage-initializer"
Aug 05 17:34:12.366931 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366677    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="storage-initializer"
Aug 05 17:34:12.366931 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366683    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kube-rbac-proxy"
Aug 05 17:34:12.366931 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366688    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kube-rbac-proxy"
Aug 05 17:34:12.366931 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366724    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kube-rbac-proxy"
Aug 05 17:34:12.366931 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366731    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="agent"
Aug 05 17:34:12.366931 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366736    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kserve-container"
Aug 05 17:34:12.366931 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366743    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f976491a-8d91-41e3-94a4-3f599c8d646f" containerName="kserve-container"
Aug 05 17:34:12.366931 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.366749    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="31da19a9-cf8c-4732-8230-c72ff5f860e8" containerName="kube-rbac-proxy"
Aug 05 17:34:12.371359 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.371341    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.373921 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.373898    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 05 17:34:12.374043 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.373971    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 05 17:34:12.374043 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.373981    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 05 17:34:12.374043 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.373992    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-992fe-kube-rbac-proxy-sar-config\""
Aug 05 17:34:12.374236 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.374047    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-992fe-predictor-serving-cert\""
Aug 05 17:34:12.374236 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.374094    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-h9rpc\""
Aug 05 17:34:12.374305 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.374282    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 17:34:12.381054 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.381030    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"]
Aug 05 17:34:12.444975 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.444895    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26c32742-280f-4bec-a639-0c2f33766777-proxy-tls\") pod \"raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.444975 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.444937    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26c32742-280f-4bec-a639-0c2f33766777-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.444975 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.444956    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26c32742-280f-4bec-a639-0c2f33766777-cabundle-cert\") pod \"raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.445243 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.445017    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26c32742-280f-4bec-a639-0c2f33766777-kserve-provision-location\") pod \"raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.445243 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.445035    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pds8g\" (UniqueName: \"kubernetes.io/projected/26c32742-280f-4bec-a639-0c2f33766777-kube-api-access-pds8g\") pod \"raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.545399 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.545361    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26c32742-280f-4bec-a639-0c2f33766777-kserve-provision-location\") pod \"raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.545399 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.545402    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pds8g\" (UniqueName: \"kubernetes.io/projected/26c32742-280f-4bec-a639-0c2f33766777-kube-api-access-pds8g\") pod \"raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.545660 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.545444    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26c32742-280f-4bec-a639-0c2f33766777-proxy-tls\") pod \"raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.545660 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.545480    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26c32742-280f-4bec-a639-0c2f33766777-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.545660 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.545505    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26c32742-280f-4bec-a639-0c2f33766777-cabundle-cert\") pod \"raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.545831 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.545805    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26c32742-280f-4bec-a639-0c2f33766777-kserve-provision-location\") pod \"raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.546165 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.546137    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26c32742-280f-4bec-a639-0c2f33766777-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.546284 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.546215    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26c32742-280f-4bec-a639-0c2f33766777-cabundle-cert\") pod \"raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.547912 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.547892    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26c32742-280f-4bec-a639-0c2f33766777-proxy-tls\") pod \"raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.554734 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.554712    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pds8g\" (UniqueName: \"kubernetes.io/projected/26c32742-280f-4bec-a639-0c2f33766777-kube-api-access-pds8g\") pod \"raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.682448 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.682399    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:12.811792 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:12.811759    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"]
Aug 05 17:34:12.814500 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:34:12.814467    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c32742_280f_4bec_a639_0c2f33766777.slice/crio-663693a7f51fab07ecad7dc07203248582c470f1b3277522107ca74882747c3e WatchSource:0}: Error finding container 663693a7f51fab07ecad7dc07203248582c470f1b3277522107ca74882747c3e: Status 404 returned error can't find the container with id 663693a7f51fab07ecad7dc07203248582c470f1b3277522107ca74882747c3e
Aug 05 17:34:13.162364 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:13.162321    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" event={"ID":"26c32742-280f-4bec-a639-0c2f33766777","Type":"ContainerStarted","Data":"855ccb14a03882febbb1035ca8dd23b4692d759036cf78dca8ce878f9dbb4e82"}
Aug 05 17:34:13.162862 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:13.162385    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" event={"ID":"26c32742-280f-4bec-a639-0c2f33766777","Type":"ContainerStarted","Data":"663693a7f51fab07ecad7dc07203248582c470f1b3277522107ca74882747c3e"}
Aug 05 17:34:17.180168 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:17.180131    2574 generic.go:356] "Generic (PLEG): container finished" podID="26c32742-280f-4bec-a639-0c2f33766777" containerID="855ccb14a03882febbb1035ca8dd23b4692d759036cf78dca8ce878f9dbb4e82" exitCode=0
Aug 05 17:34:17.180562 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:17.180173    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" event={"ID":"26c32742-280f-4bec-a639-0c2f33766777","Type":"ContainerDied","Data":"855ccb14a03882febbb1035ca8dd23b4692d759036cf78dca8ce878f9dbb4e82"}
Aug 05 17:34:18.185514 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:18.185478    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" event={"ID":"26c32742-280f-4bec-a639-0c2f33766777","Type":"ContainerStarted","Data":"d59c0ff85064f6ab09594f11a887cc985cb905e629827e6b5c25825819a854e0"}
Aug 05 17:34:18.185933 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:18.185540    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" event={"ID":"26c32742-280f-4bec-a639-0c2f33766777","Type":"ContainerStarted","Data":"a6d50368e45d8297901936d89efbf4be9462586253400b42f8ebea8a5e778472"}
Aug 05 17:34:18.185933 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:18.185864    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:18.186013 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:18.185995    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:18.187307 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:18.187288    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 05 17:34:18.209353 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:18.209298    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" podStartSLOduration=2.389905794 podStartE2EDuration="6.209279567s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.819373773s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:34:18.20772429 +0000 UTC m=+1308.499579121" watchObservedRunningTime="2026-08-05 17:34:18.209279567 +0000 UTC m=+1308.501134397"
Aug 05 17:34:19.189225 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:19.189185    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 05 17:34:20.193002 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:20.192962    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 05 17:34:25.197440 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:25.197401    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:34:25.198019 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:25.197993    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 05 17:34:35.198182 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:35.198141    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 05 17:34:45.198450 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:45.198407    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 05 17:34:55.198013 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:34:55.197922    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 05 17:35:05.198942 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:05.198898    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 05 17:35:15.198954 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:15.198904    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 05 17:35:19.252215 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.252177    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"]
Aug 05 17:35:19.255688 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.255666    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.258124 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.258101    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Aug 05 17:35:19.258244 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.258177    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Aug 05 17:35:19.267137 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.267115    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"]
Aug 05 17:35:19.296024 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.295993    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"]
Aug 05 17:35:19.299340 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.299313    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.301683 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.301659    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Aug 05 17:35:19.311560 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.311521    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"]
Aug 05 17:35:19.343449 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.343410    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5040bb41-31d4-4284-a7da-84dce23bf655-kserve-provision-location\") pod \"isvc-canary-create-predictor-7879794c64-tvsrl\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.343449 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.343453    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5040bb41-31d4-4284-a7da-84dce23bf655-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7879794c64-tvsrl\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.343697 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.343481    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5040bb41-31d4-4284-a7da-84dce23bf655-cabundle-cert\") pod \"isvc-canary-create-predictor-7879794c64-tvsrl\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.343697 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.343514    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lc6k\" (UniqueName: \"kubernetes.io/projected/5040bb41-31d4-4284-a7da-84dce23bf655-kube-api-access-6lc6k\") pod \"isvc-canary-create-predictor-7879794c64-tvsrl\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.343697 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.343552    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5040bb41-31d4-4284-a7da-84dce23bf655-proxy-tls\") pod \"isvc-canary-create-predictor-7879794c64-tvsrl\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.444849 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.444814    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5040bb41-31d4-4284-a7da-84dce23bf655-kserve-provision-location\") pod \"isvc-canary-create-predictor-7879794c64-tvsrl\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.444849 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.444852    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5040bb41-31d4-4284-a7da-84dce23bf655-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7879794c64-tvsrl\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.445087 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.444872    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5040bb41-31d4-4284-a7da-84dce23bf655-cabundle-cert\") pod \"isvc-canary-create-predictor-7879794c64-tvsrl\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.445087 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.444896    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lc6k\" (UniqueName: \"kubernetes.io/projected/5040bb41-31d4-4284-a7da-84dce23bf655-kube-api-access-6lc6k\") pod \"isvc-canary-create-predictor-7879794c64-tvsrl\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.445087 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.444923    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-proxy-tls\") pod \"isvc-canary-create-v2-predictor-59c98df545-5pwj5\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.445087 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.445047    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5040bb41-31d4-4284-a7da-84dce23bf655-proxy-tls\") pod \"isvc-canary-create-predictor-7879794c64-tvsrl\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.445300 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.445127    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-59c98df545-5pwj5\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.445300 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.445155    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glkb7\" (UniqueName: \"kubernetes.io/projected/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kube-api-access-glkb7\") pod \"isvc-canary-create-v2-predictor-59c98df545-5pwj5\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.445300 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.445174    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-59c98df545-5pwj5\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.445300 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.445250    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-59c98df545-5pwj5\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.445300 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.445294    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5040bb41-31d4-4284-a7da-84dce23bf655-kserve-provision-location\") pod \"isvc-canary-create-predictor-7879794c64-tvsrl\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.445714 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.445690    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5040bb41-31d4-4284-a7da-84dce23bf655-cabundle-cert\") pod \"isvc-canary-create-predictor-7879794c64-tvsrl\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.445763 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.445690    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5040bb41-31d4-4284-a7da-84dce23bf655-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7879794c64-tvsrl\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.447596 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.447573    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5040bb41-31d4-4284-a7da-84dce23bf655-proxy-tls\") pod \"isvc-canary-create-predictor-7879794c64-tvsrl\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.454614 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.454592    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lc6k\" (UniqueName: \"kubernetes.io/projected/5040bb41-31d4-4284-a7da-84dce23bf655-kube-api-access-6lc6k\") pod \"isvc-canary-create-predictor-7879794c64-tvsrl\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.546440 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.546340    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glkb7\" (UniqueName: \"kubernetes.io/projected/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kube-api-access-glkb7\") pod \"isvc-canary-create-v2-predictor-59c98df545-5pwj5\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.546440 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.546392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-59c98df545-5pwj5\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.546670 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.546455    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-59c98df545-5pwj5\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.546670 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.546602    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-proxy-tls\") pod \"isvc-canary-create-v2-predictor-59c98df545-5pwj5\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.546670 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.546662    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-59c98df545-5pwj5\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.547104 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.547056    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-59c98df545-5pwj5\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.547337 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.547316    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-59c98df545-5pwj5\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.547396 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.547324    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-59c98df545-5pwj5\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.549126 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.549098    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-proxy-tls\") pod \"isvc-canary-create-v2-predictor-59c98df545-5pwj5\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.556967 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.556933    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glkb7\" (UniqueName: \"kubernetes.io/projected/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kube-api-access-glkb7\") pod \"isvc-canary-create-v2-predictor-59c98df545-5pwj5\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.565616 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.565595    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:19.617263 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.617198    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:19.704443 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.704411    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"]
Aug 05 17:35:19.706991 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:35:19.706965    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5040bb41_31d4_4284_a7da_84dce23bf655.slice/crio-f351619be3597cc339c6ef3035446c6cd8ab6658bc77ab684c1a481af84654d8 WatchSource:0}: Error finding container f351619be3597cc339c6ef3035446c6cd8ab6658bc77ab684c1a481af84654d8: Status 404 returned error can't find the container with id f351619be3597cc339c6ef3035446c6cd8ab6658bc77ab684c1a481af84654d8
Aug 05 17:35:19.770866 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:19.770845    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"]
Aug 05 17:35:19.773170 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:35:19.773145    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6ec46a_cb12_48d5_ab87_46be6b6eb481.slice/crio-510b355e3706d9c5c1b0e0be0154bd6adf6c77e2843c58af91c21f100eb61d71 WatchSource:0}: Error finding container 510b355e3706d9c5c1b0e0be0154bd6adf6c77e2843c58af91c21f100eb61d71: Status 404 returned error can't find the container with id 510b355e3706d9c5c1b0e0be0154bd6adf6c77e2843c58af91c21f100eb61d71
Aug 05 17:35:20.409629 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:20.409590    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" event={"ID":"fa6ec46a-cb12-48d5-ab87-46be6b6eb481","Type":"ContainerStarted","Data":"13610eb97f44a74c6d57373611866ffcd27f0a7c8259f389c9c8e45ab4b57bb5"}
Aug 05 17:35:20.410099 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:20.409661    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" event={"ID":"fa6ec46a-cb12-48d5-ab87-46be6b6eb481","Type":"ContainerStarted","Data":"510b355e3706d9c5c1b0e0be0154bd6adf6c77e2843c58af91c21f100eb61d71"}
Aug 05 17:35:20.411221 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:20.411194    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" event={"ID":"5040bb41-31d4-4284-a7da-84dce23bf655","Type":"ContainerStarted","Data":"5aba46e5ef91a47814d2fb4df26f79a2ea2beaefd1d872ab4ef9b9977e0e5691"}
Aug 05 17:35:20.411309 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:20.411254    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" event={"ID":"5040bb41-31d4-4284-a7da-84dce23bf655","Type":"ContainerStarted","Data":"f351619be3597cc339c6ef3035446c6cd8ab6658bc77ab684c1a481af84654d8"}
Aug 05 17:35:24.428306 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:24.428266    2574 generic.go:356] "Generic (PLEG): container finished" podID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerID="13610eb97f44a74c6d57373611866ffcd27f0a7c8259f389c9c8e45ab4b57bb5" exitCode=0
Aug 05 17:35:24.428751 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:24.428340    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" event={"ID":"fa6ec46a-cb12-48d5-ab87-46be6b6eb481","Type":"ContainerDied","Data":"13610eb97f44a74c6d57373611866ffcd27f0a7c8259f389c9c8e45ab4b57bb5"}
Aug 05 17:35:24.429919 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:24.429899    2574 generic.go:356] "Generic (PLEG): container finished" podID="5040bb41-31d4-4284-a7da-84dce23bf655" containerID="5aba46e5ef91a47814d2fb4df26f79a2ea2beaefd1d872ab4ef9b9977e0e5691" exitCode=0
Aug 05 17:35:24.430003 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:24.429974    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" event={"ID":"5040bb41-31d4-4284-a7da-84dce23bf655","Type":"ContainerDied","Data":"5aba46e5ef91a47814d2fb4df26f79a2ea2beaefd1d872ab4ef9b9977e0e5691"}
Aug 05 17:35:25.198654 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:25.198609    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 05 17:35:25.436108 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:25.436057    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" event={"ID":"5040bb41-31d4-4284-a7da-84dce23bf655","Type":"ContainerStarted","Data":"4942b4d4622045c7ded037ee01ff435c7d609e90a7a7365fd03221521c0f8d06"}
Aug 05 17:35:25.436591 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:25.436165    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" event={"ID":"5040bb41-31d4-4284-a7da-84dce23bf655","Type":"ContainerStarted","Data":"b39525bdd73827a4d8ffa43e445530d9d335d6ac3265a92c808e6fbb13e39235"}
Aug 05 17:35:25.436591 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:25.436356    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:25.438219 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:25.438177    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" event={"ID":"fa6ec46a-cb12-48d5-ab87-46be6b6eb481","Type":"ContainerStarted","Data":"39372092be2f1facefd4dbc6025c3fa8e1fbeb2cb02e5a3296f326aabc06f024"}
Aug 05 17:35:25.438333 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:25.438281    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" event={"ID":"fa6ec46a-cb12-48d5-ab87-46be6b6eb481","Type":"ContainerStarted","Data":"b0cee98236324e2f3cc4e6da4c1fc50e37bdae822a7a272f69fc616e31def132"}
Aug 05 17:35:25.438472 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:25.438456    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:25.457888 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:25.457798    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" podStartSLOduration=2.622683382 podStartE2EDuration="6.457781804s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.835098422s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:35:25.456241453 +0000 UTC m=+1375.748096284" watchObservedRunningTime="2026-08-05 17:35:25.457781804 +0000 UTC m=+1375.749636635"
Aug 05 17:35:25.476136 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:25.476062    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" podStartSLOduration=2.760249501 podStartE2EDuration="6.476049202s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.715799701s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:35:25.475816999 +0000 UTC m=+1375.767671833" watchObservedRunningTime="2026-08-05 17:35:25.476049202 +0000 UTC m=+1375.767904034"
Aug 05 17:35:26.442127 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:26.442092    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:26.442524 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:26.442160    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:26.443163 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:26.443135    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 05 17:35:26.443262 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:26.443170    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 05 17:35:27.446159 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:27.446116    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 05 17:35:27.446558 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:27.446247    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 05 17:35:32.450980 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:32.450924    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:35:32.451670 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:32.451633    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:35:32.451670 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:32.451638    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 05 17:35:32.452227 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:32.452196    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 05 17:35:35.199147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:35.199114    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:35:42.452115 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:42.452053    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 05 17:35:42.452664 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:42.452229    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 05 17:35:42.473736 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:42.473703    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"]
Aug 05 17:35:42.473958 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:42.473933    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kserve-container" containerID="cri-o://a6d50368e45d8297901936d89efbf4be9462586253400b42f8ebea8a5e778472" gracePeriod=30
Aug 05 17:35:42.474042 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:42.473994    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kube-rbac-proxy" containerID="cri-o://d59c0ff85064f6ab09594f11a887cc985cb905e629827e6b5c25825819a854e0" gracePeriod=30
Aug 05 17:35:43.505374 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:43.505342    2574 generic.go:356] "Generic (PLEG): container finished" podID="26c32742-280f-4bec-a639-0c2f33766777" containerID="d59c0ff85064f6ab09594f11a887cc985cb905e629827e6b5c25825819a854e0" exitCode=2
Aug 05 17:35:43.505753 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:43.505392    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" event={"ID":"26c32742-280f-4bec-a639-0c2f33766777","Type":"ContainerDied","Data":"d59c0ff85064f6ab09594f11a887cc985cb905e629827e6b5c25825819a854e0"}
Aug 05 17:35:45.193798 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:45.193749    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Aug 05 17:35:45.198157 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:45.198123    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 05 17:35:47.722415 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.722390    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:35:47.761161 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.761126    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26c32742-280f-4bec-a639-0c2f33766777-kube-rbac-proxy-sar-config\") pod \"26c32742-280f-4bec-a639-0c2f33766777\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") "
Aug 05 17:35:47.761343 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.761170    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26c32742-280f-4bec-a639-0c2f33766777-cabundle-cert\") pod \"26c32742-280f-4bec-a639-0c2f33766777\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") "
Aug 05 17:35:47.761343 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.761206    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26c32742-280f-4bec-a639-0c2f33766777-proxy-tls\") pod \"26c32742-280f-4bec-a639-0c2f33766777\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") "
Aug 05 17:35:47.761343 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.761236    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pds8g\" (UniqueName: \"kubernetes.io/projected/26c32742-280f-4bec-a639-0c2f33766777-kube-api-access-pds8g\") pod \"26c32742-280f-4bec-a639-0c2f33766777\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") "
Aug 05 17:35:47.761343 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.761254    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26c32742-280f-4bec-a639-0c2f33766777-kserve-provision-location\") pod \"26c32742-280f-4bec-a639-0c2f33766777\" (UID: \"26c32742-280f-4bec-a639-0c2f33766777\") "
Aug 05 17:35:47.761672 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.761544    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c32742-280f-4bec-a639-0c2f33766777-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26c32742-280f-4bec-a639-0c2f33766777" (UID: "26c32742-280f-4bec-a639-0c2f33766777"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:35:47.761672 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.761662    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c32742-280f-4bec-a639-0c2f33766777-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26c32742-280f-4bec-a639-0c2f33766777" (UID: "26c32742-280f-4bec-a639-0c2f33766777"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:35:47.761860 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.761665    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c32742-280f-4bec-a639-0c2f33766777-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26c32742-280f-4bec-a639-0c2f33766777" (UID: "26c32742-280f-4bec-a639-0c2f33766777"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:35:47.763414 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.763386    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c32742-280f-4bec-a639-0c2f33766777-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26c32742-280f-4bec-a639-0c2f33766777" (UID: "26c32742-280f-4bec-a639-0c2f33766777"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:35:47.763511 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.763479    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c32742-280f-4bec-a639-0c2f33766777-kube-api-access-pds8g" (OuterVolumeSpecName: "kube-api-access-pds8g") pod "26c32742-280f-4bec-a639-0c2f33766777" (UID: "26c32742-280f-4bec-a639-0c2f33766777"). InnerVolumeSpecName "kube-api-access-pds8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:35:47.862215 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.862161    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26c32742-280f-4bec-a639-0c2f33766777-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:35:47.862215 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.862209    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pds8g\" (UniqueName: \"kubernetes.io/projected/26c32742-280f-4bec-a639-0c2f33766777-kube-api-access-pds8g\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:35:47.862215 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.862220    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26c32742-280f-4bec-a639-0c2f33766777-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:35:47.862215 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.862229    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26c32742-280f-4bec-a639-0c2f33766777-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:35:47.862500 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:47.862240    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26c32742-280f-4bec-a639-0c2f33766777-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:35:48.527192 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:48.527159    2574 generic.go:356] "Generic (PLEG): container finished" podID="26c32742-280f-4bec-a639-0c2f33766777" containerID="a6d50368e45d8297901936d89efbf4be9462586253400b42f8ebea8a5e778472" exitCode=0
Aug 05 17:35:48.527395 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:48.527201    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" event={"ID":"26c32742-280f-4bec-a639-0c2f33766777","Type":"ContainerDied","Data":"a6d50368e45d8297901936d89efbf4be9462586253400b42f8ebea8a5e778472"}
Aug 05 17:35:48.527395 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:48.527242    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"
Aug 05 17:35:48.527395 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:48.527266    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4" event={"ID":"26c32742-280f-4bec-a639-0c2f33766777","Type":"ContainerDied","Data":"663693a7f51fab07ecad7dc07203248582c470f1b3277522107ca74882747c3e"}
Aug 05 17:35:48.527395 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:48.527293    2574 scope.go:117] "RemoveContainer" containerID="d59c0ff85064f6ab09594f11a887cc985cb905e629827e6b5c25825819a854e0"
Aug 05 17:35:48.535247 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:48.535229    2574 scope.go:117] "RemoveContainer" containerID="a6d50368e45d8297901936d89efbf4be9462586253400b42f8ebea8a5e778472"
Aug 05 17:35:48.542005 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:48.541989    2574 scope.go:117] "RemoveContainer" containerID="855ccb14a03882febbb1035ca8dd23b4692d759036cf78dca8ce878f9dbb4e82"
Aug 05 17:35:48.548682 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:48.548658    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"]
Aug 05 17:35:48.549844 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:48.549823    2574 scope.go:117] "RemoveContainer" containerID="d59c0ff85064f6ab09594f11a887cc985cb905e629827e6b5c25825819a854e0"
Aug 05 17:35:48.550104 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:35:48.550081    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59c0ff85064f6ab09594f11a887cc985cb905e629827e6b5c25825819a854e0\": container with ID starting with d59c0ff85064f6ab09594f11a887cc985cb905e629827e6b5c25825819a854e0 not found: ID does not exist" containerID="d59c0ff85064f6ab09594f11a887cc985cb905e629827e6b5c25825819a854e0"
Aug 05 17:35:48.550162 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:48.550118    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59c0ff85064f6ab09594f11a887cc985cb905e629827e6b5c25825819a854e0"} err="failed to get container status \"d59c0ff85064f6ab09594f11a887cc985cb905e629827e6b5c25825819a854e0\": rpc error: code = NotFound desc = could not find container \"d59c0ff85064f6ab09594f11a887cc985cb905e629827e6b5c25825819a854e0\": container with ID starting with d59c0ff85064f6ab09594f11a887cc985cb905e629827e6b5c25825819a854e0 not found: ID does not exist"
Aug 05 17:35:48.550162 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:48.550144    2574 scope.go:117] "RemoveContainer" containerID="a6d50368e45d8297901936d89efbf4be9462586253400b42f8ebea8a5e778472"
Aug 05 17:35:48.550363 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:35:48.550347    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d50368e45d8297901936d89efbf4be9462586253400b42f8ebea8a5e778472\": container with ID starting with a6d50368e45d8297901936d89efbf4be9462586253400b42f8ebea8a5e778472 not found: ID does not exist" containerID="a6d50368e45d8297901936d89efbf4be9462586253400b42f8ebea8a5e778472"
Aug 05 17:35:48.550407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:48.550369    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d50368e45d8297901936d89efbf4be9462586253400b42f8ebea8a5e778472"} err="failed to get container status \"a6d50368e45d8297901936d89efbf4be9462586253400b42f8ebea8a5e778472\": rpc error: code = NotFound desc = could not find container \"a6d50368e45d8297901936d89efbf4be9462586253400b42f8ebea8a5e778472\": container with ID starting with a6d50368e45d8297901936d89efbf4be9462586253400b42f8ebea8a5e778472 not found: ID does not exist"
Aug 05 17:35:48.550407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:48.550385    2574 scope.go:117] "RemoveContainer" containerID="855ccb14a03882febbb1035ca8dd23b4692d759036cf78dca8ce878f9dbb4e82"
Aug 05 17:35:48.550612 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:35:48.550594    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"855ccb14a03882febbb1035ca8dd23b4692d759036cf78dca8ce878f9dbb4e82\": container with ID starting with 855ccb14a03882febbb1035ca8dd23b4692d759036cf78dca8ce878f9dbb4e82 not found: ID does not exist" containerID="855ccb14a03882febbb1035ca8dd23b4692d759036cf78dca8ce878f9dbb4e82"
Aug 05 17:35:48.550708 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:48.550622    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"855ccb14a03882febbb1035ca8dd23b4692d759036cf78dca8ce878f9dbb4e82"} err="failed to get container status \"855ccb14a03882febbb1035ca8dd23b4692d759036cf78dca8ce878f9dbb4e82\": rpc error: code = NotFound desc = could not find container \"855ccb14a03882febbb1035ca8dd23b4692d759036cf78dca8ce878f9dbb4e82\": container with ID starting with 855ccb14a03882febbb1035ca8dd23b4692d759036cf78dca8ce878f9dbb4e82 not found: ID does not exist"
Aug 05 17:35:48.554038 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:48.554017    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-992fe-predictor-6d7b9fcc49-dn7b4"]
Aug 05 17:35:50.194997 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:50.194963    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26c32742-280f-4bec-a639-0c2f33766777" path="/var/lib/kubelet/pods/26c32742-280f-4bec-a639-0c2f33766777/volumes"
Aug 05 17:35:52.452326 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:52.452289    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 05 17:35:52.452739 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:35:52.452281    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 05 17:36:02.452513 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:02.452468    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 05 17:36:02.452936 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:02.452472    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 05 17:36:12.451657 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:12.451617    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 05 17:36:12.452153 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:12.452125    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 05 17:36:22.452400 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:22.452310    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 05 17:36:22.452857 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:22.452308    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 05 17:36:32.452239 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:32.452197    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 05 17:36:32.452634 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:32.452200    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 05 17:36:37.189907 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:37.189858    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 05 17:36:42.452743 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:42.452714    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:36:47.190705 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:47.190674    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:36:49.341156 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:49.341119    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"]
Aug 05 17:36:49.341621 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:49.341380    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kserve-container" containerID="cri-o://b39525bdd73827a4d8ffa43e445530d9d335d6ac3265a92c808e6fbb13e39235" gracePeriod=30
Aug 05 17:36:49.341621 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:49.341418    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kube-rbac-proxy" containerID="cri-o://4942b4d4622045c7ded037ee01ff435c7d609e90a7a7365fd03221521c0f8d06" gracePeriod=30
Aug 05 17:36:49.345438 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:49.345412    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"]
Aug 05 17:36:49.345671 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:49.345635    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kserve-container" containerID="cri-o://b0cee98236324e2f3cc4e6da4c1fc50e37bdae822a7a272f69fc616e31def132" gracePeriod=30
Aug 05 17:36:49.345754 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:49.345703    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kube-rbac-proxy" containerID="cri-o://39372092be2f1facefd4dbc6025c3fa8e1fbeb2cb02e5a3296f326aabc06f024" gracePeriod=30
Aug 05 17:36:49.748855 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:49.748823    2574 generic.go:356] "Generic (PLEG): container finished" podID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerID="39372092be2f1facefd4dbc6025c3fa8e1fbeb2cb02e5a3296f326aabc06f024" exitCode=2
Aug 05 17:36:49.749047 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:49.748902    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" event={"ID":"fa6ec46a-cb12-48d5-ab87-46be6b6eb481","Type":"ContainerDied","Data":"39372092be2f1facefd4dbc6025c3fa8e1fbeb2cb02e5a3296f326aabc06f024"}
Aug 05 17:36:49.750851 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:49.750833    2574 generic.go:356] "Generic (PLEG): container finished" podID="5040bb41-31d4-4284-a7da-84dce23bf655" containerID="4942b4d4622045c7ded037ee01ff435c7d609e90a7a7365fd03221521c0f8d06" exitCode=2
Aug 05 17:36:49.750950 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:49.750866    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" event={"ID":"5040bb41-31d4-4284-a7da-84dce23bf655","Type":"ContainerDied","Data":"4942b4d4622045c7ded037ee01ff435c7d609e90a7a7365fd03221521c0f8d06"}
Aug 05 17:36:52.446872 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:52.446823    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Aug 05 17:36:52.447297 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:52.446823    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Aug 05 17:36:52.452288 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:52.452264    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 05 17:36:54.394401 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.394373    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:36:54.485413 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.485384    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5040bb41-31d4-4284-a7da-84dce23bf655-kserve-provision-location\") pod \"5040bb41-31d4-4284-a7da-84dce23bf655\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") "
Aug 05 17:36:54.485533 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.485428    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5040bb41-31d4-4284-a7da-84dce23bf655-proxy-tls\") pod \"5040bb41-31d4-4284-a7da-84dce23bf655\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") "
Aug 05 17:36:54.485533 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.485455    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5040bb41-31d4-4284-a7da-84dce23bf655-cabundle-cert\") pod \"5040bb41-31d4-4284-a7da-84dce23bf655\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") "
Aug 05 17:36:54.485533 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.485489    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5040bb41-31d4-4284-a7da-84dce23bf655-kube-rbac-proxy-sar-config\") pod \"5040bb41-31d4-4284-a7da-84dce23bf655\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") "
Aug 05 17:36:54.485533 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.485513    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lc6k\" (UniqueName: \"kubernetes.io/projected/5040bb41-31d4-4284-a7da-84dce23bf655-kube-api-access-6lc6k\") pod \"5040bb41-31d4-4284-a7da-84dce23bf655\" (UID: \"5040bb41-31d4-4284-a7da-84dce23bf655\") "
Aug 05 17:36:54.485790 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.485688    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5040bb41-31d4-4284-a7da-84dce23bf655-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5040bb41-31d4-4284-a7da-84dce23bf655" (UID: "5040bb41-31d4-4284-a7da-84dce23bf655"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:36:54.485908 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.485825    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5040bb41-31d4-4284-a7da-84dce23bf655-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5040bb41-31d4-4284-a7da-84dce23bf655" (UID: "5040bb41-31d4-4284-a7da-84dce23bf655"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:36:54.485908 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.485859    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5040bb41-31d4-4284-a7da-84dce23bf655-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5040bb41-31d4-4284-a7da-84dce23bf655" (UID: "5040bb41-31d4-4284-a7da-84dce23bf655"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:36:54.487670 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.487642    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5040bb41-31d4-4284-a7da-84dce23bf655-kube-api-access-6lc6k" (OuterVolumeSpecName: "kube-api-access-6lc6k") pod "5040bb41-31d4-4284-a7da-84dce23bf655" (UID: "5040bb41-31d4-4284-a7da-84dce23bf655"). InnerVolumeSpecName "kube-api-access-6lc6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:36:54.487811 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.487687    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5040bb41-31d4-4284-a7da-84dce23bf655-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5040bb41-31d4-4284-a7da-84dce23bf655" (UID: "5040bb41-31d4-4284-a7da-84dce23bf655"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:36:54.505621 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.505594    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:36:54.545323 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545281    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"]
Aug 05 17:36:54.545666 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545647    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="storage-initializer"
Aug 05 17:36:54.545731 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545670    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="storage-initializer"
Aug 05 17:36:54.545731 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545693    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kube-rbac-proxy"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545738    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kube-rbac-proxy"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545750    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="storage-initializer"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545760    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="storage-initializer"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545795    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kserve-container"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545803    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kserve-container"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545813    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kube-rbac-proxy"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545821    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kube-rbac-proxy"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545846    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kube-rbac-proxy"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545855    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kube-rbac-proxy"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545866    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kserve-container"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545876    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kserve-container"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545889    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="storage-initializer"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545897    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="storage-initializer"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545906    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kserve-container"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545914    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kserve-container"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.545998    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kserve-container"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.546016    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" containerName="kube-rbac-proxy"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.546029    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kserve-container"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.546039    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kube-rbac-proxy"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.546049    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c32742-280f-4bec-a639-0c2f33766777" containerName="kube-rbac-proxy"
Aug 05 17:36:54.546109 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.546059    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerName="kserve-container"
Aug 05 17:36:54.549226 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.549210    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:54.551718 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.551696    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Aug 05 17:36:54.551823 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.551765    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Aug 05 17:36:54.561740 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.561716    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"]
Aug 05 17:36:54.586174 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586140    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-cabundle-cert\") pod \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") "
Aug 05 17:36:54.586174 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586178    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glkb7\" (UniqueName: \"kubernetes.io/projected/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kube-api-access-glkb7\") pod \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") "
Aug 05 17:36:54.586399 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586220    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-proxy-tls\") pod \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") "
Aug 05 17:36:54.586399 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586252    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kserve-provision-location\") pod \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") "
Aug 05 17:36:54.586399 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586290    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kube-rbac-proxy-sar-config\") pod \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\" (UID: \"fa6ec46a-cb12-48d5-ab87-46be6b6eb481\") "
Aug 05 17:36:54.586536 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586395    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/983a04c7-ea54-4af8-bcb5-dc449a407838-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:54.586536 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586435    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvfkm\" (UniqueName: \"kubernetes.io/projected/983a04c7-ea54-4af8-bcb5-dc449a407838-kube-api-access-rvfkm\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:54.586536 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586463    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/983a04c7-ea54-4af8-bcb5-dc449a407838-cabundle-cert\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:54.586683 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586546    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/983a04c7-ea54-4af8-bcb5-dc449a407838-proxy-tls\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:54.586683 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586571    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa6ec46a-cb12-48d5-ab87-46be6b6eb481" (UID: "fa6ec46a-cb12-48d5-ab87-46be6b6eb481"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:36:54.586683 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586603    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa6ec46a-cb12-48d5-ab87-46be6b6eb481" (UID: "fa6ec46a-cb12-48d5-ab87-46be6b6eb481"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:36:54.586683 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586604    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/983a04c7-ea54-4af8-bcb5-dc449a407838-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:54.586683 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586660    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa6ec46a-cb12-48d5-ab87-46be6b6eb481" (UID: "fa6ec46a-cb12-48d5-ab87-46be6b6eb481"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:36:54.586878 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586708    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5040bb41-31d4-4284-a7da-84dce23bf655-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:36:54.586878 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586719    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:36:54.586878 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586728    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5040bb41-31d4-4284-a7da-84dce23bf655-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:36:54.586878 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586747    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5040bb41-31d4-4284-a7da-84dce23bf655-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:36:54.586878 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586757    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5040bb41-31d4-4284-a7da-84dce23bf655-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:36:54.586878 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586766    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lc6k\" (UniqueName: \"kubernetes.io/projected/5040bb41-31d4-4284-a7da-84dce23bf655-kube-api-access-6lc6k\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:36:54.586878 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586775    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:36:54.586878 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.586783    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:36:54.588351 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.588332    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kube-api-access-glkb7" (OuterVolumeSpecName: "kube-api-access-glkb7") pod "fa6ec46a-cb12-48d5-ab87-46be6b6eb481" (UID: "fa6ec46a-cb12-48d5-ab87-46be6b6eb481"). InnerVolumeSpecName "kube-api-access-glkb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:36:54.588403 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.588357    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa6ec46a-cb12-48d5-ab87-46be6b6eb481" (UID: "fa6ec46a-cb12-48d5-ab87-46be6b6eb481"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:36:54.687887 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.687849    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/983a04c7-ea54-4af8-bcb5-dc449a407838-proxy-tls\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:54.687887 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.687890    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/983a04c7-ea54-4af8-bcb5-dc449a407838-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:54.688157 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.687939    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/983a04c7-ea54-4af8-bcb5-dc449a407838-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:54.688157 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.687960    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvfkm\" (UniqueName: \"kubernetes.io/projected/983a04c7-ea54-4af8-bcb5-dc449a407838-kube-api-access-rvfkm\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:54.688157 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.687978    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/983a04c7-ea54-4af8-bcb5-dc449a407838-cabundle-cert\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:54.688157 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.688004    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glkb7\" (UniqueName: \"kubernetes.io/projected/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-kube-api-access-glkb7\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:36:54.688157 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.688015    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa6ec46a-cb12-48d5-ab87-46be6b6eb481-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:36:54.688157 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:36:54.688031    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Aug 05 17:36:54.688157 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:36:54.688130    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/983a04c7-ea54-4af8-bcb5-dc449a407838-proxy-tls podName:983a04c7-ea54-4af8-bcb5-dc449a407838 nodeName:}" failed. No retries permitted until 2026-08-05 17:36:55.188106955 +0000 UTC m=+1465.479961763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/983a04c7-ea54-4af8-bcb5-dc449a407838-proxy-tls") pod "isvc-canary-promote-predictor-5cc67bffb8-m4ztq" (UID: "983a04c7-ea54-4af8-bcb5-dc449a407838") : secret "isvc-canary-promote-predictor-serving-cert" not found
Aug 05 17:36:54.688538 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.688363    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/983a04c7-ea54-4af8-bcb5-dc449a407838-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:54.688665 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.688648    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/983a04c7-ea54-4af8-bcb5-dc449a407838-cabundle-cert\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:54.688701 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.688663    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/983a04c7-ea54-4af8-bcb5-dc449a407838-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:54.700752 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.700727    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvfkm\" (UniqueName: \"kubernetes.io/projected/983a04c7-ea54-4af8-bcb5-dc449a407838-kube-api-access-rvfkm\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:54.771552 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.771518    2574 generic.go:356] "Generic (PLEG): container finished" podID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" containerID="b0cee98236324e2f3cc4e6da4c1fc50e37bdae822a7a272f69fc616e31def132" exitCode=0
Aug 05 17:36:54.771717 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.771574    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"
Aug 05 17:36:54.771717 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.771594    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" event={"ID":"fa6ec46a-cb12-48d5-ab87-46be6b6eb481","Type":"ContainerDied","Data":"b0cee98236324e2f3cc4e6da4c1fc50e37bdae822a7a272f69fc616e31def132"}
Aug 05 17:36:54.771717 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.771663    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5" event={"ID":"fa6ec46a-cb12-48d5-ab87-46be6b6eb481","Type":"ContainerDied","Data":"510b355e3706d9c5c1b0e0be0154bd6adf6c77e2843c58af91c21f100eb61d71"}
Aug 05 17:36:54.771717 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.771701    2574 scope.go:117] "RemoveContainer" containerID="39372092be2f1facefd4dbc6025c3fa8e1fbeb2cb02e5a3296f326aabc06f024"
Aug 05 17:36:54.773815 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.773792    2574 generic.go:356] "Generic (PLEG): container finished" podID="5040bb41-31d4-4284-a7da-84dce23bf655" containerID="b39525bdd73827a4d8ffa43e445530d9d335d6ac3265a92c808e6fbb13e39235" exitCode=0
Aug 05 17:36:54.773920 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.773826    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" event={"ID":"5040bb41-31d4-4284-a7da-84dce23bf655","Type":"ContainerDied","Data":"b39525bdd73827a4d8ffa43e445530d9d335d6ac3265a92c808e6fbb13e39235"}
Aug 05 17:36:54.773920 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.773845    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"
Aug 05 17:36:54.773920 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.773875    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl" event={"ID":"5040bb41-31d4-4284-a7da-84dce23bf655","Type":"ContainerDied","Data":"f351619be3597cc339c6ef3035446c6cd8ab6658bc77ab684c1a481af84654d8"}
Aug 05 17:36:54.780658 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.780635    2574 scope.go:117] "RemoveContainer" containerID="b0cee98236324e2f3cc4e6da4c1fc50e37bdae822a7a272f69fc616e31def132"
Aug 05 17:36:54.788724 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.788705    2574 scope.go:117] "RemoveContainer" containerID="13610eb97f44a74c6d57373611866ffcd27f0a7c8259f389c9c8e45ab4b57bb5"
Aug 05 17:36:54.796738 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.796522    2574 scope.go:117] "RemoveContainer" containerID="39372092be2f1facefd4dbc6025c3fa8e1fbeb2cb02e5a3296f326aabc06f024"
Aug 05 17:36:54.797148 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:36:54.797123    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39372092be2f1facefd4dbc6025c3fa8e1fbeb2cb02e5a3296f326aabc06f024\": container with ID starting with 39372092be2f1facefd4dbc6025c3fa8e1fbeb2cb02e5a3296f326aabc06f024 not found: ID does not exist" containerID="39372092be2f1facefd4dbc6025c3fa8e1fbeb2cb02e5a3296f326aabc06f024"
Aug 05 17:36:54.797269 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.797160    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39372092be2f1facefd4dbc6025c3fa8e1fbeb2cb02e5a3296f326aabc06f024"} err="failed to get container status \"39372092be2f1facefd4dbc6025c3fa8e1fbeb2cb02e5a3296f326aabc06f024\": rpc error: code = NotFound desc = could not find container \"39372092be2f1facefd4dbc6025c3fa8e1fbeb2cb02e5a3296f326aabc06f024\": container with ID starting with 39372092be2f1facefd4dbc6025c3fa8e1fbeb2cb02e5a3296f326aabc06f024 not found: ID does not exist"
Aug 05 17:36:54.797269 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.797186    2574 scope.go:117] "RemoveContainer" containerID="b0cee98236324e2f3cc4e6da4c1fc50e37bdae822a7a272f69fc616e31def132"
Aug 05 17:36:54.797513 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:36:54.797485    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0cee98236324e2f3cc4e6da4c1fc50e37bdae822a7a272f69fc616e31def132\": container with ID starting with b0cee98236324e2f3cc4e6da4c1fc50e37bdae822a7a272f69fc616e31def132 not found: ID does not exist" containerID="b0cee98236324e2f3cc4e6da4c1fc50e37bdae822a7a272f69fc616e31def132"
Aug 05 17:36:54.797568 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.797522    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0cee98236324e2f3cc4e6da4c1fc50e37bdae822a7a272f69fc616e31def132"} err="failed to get container status \"b0cee98236324e2f3cc4e6da4c1fc50e37bdae822a7a272f69fc616e31def132\": rpc error: code = NotFound desc = could not find container \"b0cee98236324e2f3cc4e6da4c1fc50e37bdae822a7a272f69fc616e31def132\": container with ID starting with b0cee98236324e2f3cc4e6da4c1fc50e37bdae822a7a272f69fc616e31def132 not found: ID does not exist"
Aug 05 17:36:54.797568 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.797551    2574 scope.go:117] "RemoveContainer" containerID="13610eb97f44a74c6d57373611866ffcd27f0a7c8259f389c9c8e45ab4b57bb5"
Aug 05 17:36:54.797809 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:36:54.797791    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13610eb97f44a74c6d57373611866ffcd27f0a7c8259f389c9c8e45ab4b57bb5\": container with ID starting with 13610eb97f44a74c6d57373611866ffcd27f0a7c8259f389c9c8e45ab4b57bb5 not found: ID does not exist" containerID="13610eb97f44a74c6d57373611866ffcd27f0a7c8259f389c9c8e45ab4b57bb5"
Aug 05 17:36:54.797880 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.797819    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13610eb97f44a74c6d57373611866ffcd27f0a7c8259f389c9c8e45ab4b57bb5"} err="failed to get container status \"13610eb97f44a74c6d57373611866ffcd27f0a7c8259f389c9c8e45ab4b57bb5\": rpc error: code = NotFound desc = could not find container \"13610eb97f44a74c6d57373611866ffcd27f0a7c8259f389c9c8e45ab4b57bb5\": container with ID starting with 13610eb97f44a74c6d57373611866ffcd27f0a7c8259f389c9c8e45ab4b57bb5 not found: ID does not exist"
Aug 05 17:36:54.797880 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.797840    2574 scope.go:117] "RemoveContainer" containerID="4942b4d4622045c7ded037ee01ff435c7d609e90a7a7365fd03221521c0f8d06"
Aug 05 17:36:54.798855 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.798837    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"]
Aug 05 17:36:54.805906 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.805839    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59c98df545-5pwj5"]
Aug 05 17:36:54.805964 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.805943    2574 scope.go:117] "RemoveContainer" containerID="b39525bdd73827a4d8ffa43e445530d9d335d6ac3265a92c808e6fbb13e39235"
Aug 05 17:36:54.813859 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.813835    2574 scope.go:117] "RemoveContainer" containerID="5aba46e5ef91a47814d2fb4df26f79a2ea2beaefd1d872ab4ef9b9977e0e5691"
Aug 05 17:36:54.818120 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.818097    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"]
Aug 05 17:36:54.820927 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.820907    2574 scope.go:117] "RemoveContainer" containerID="4942b4d4622045c7ded037ee01ff435c7d609e90a7a7365fd03221521c0f8d06"
Aug 05 17:36:54.821453 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:36:54.821413    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4942b4d4622045c7ded037ee01ff435c7d609e90a7a7365fd03221521c0f8d06\": container with ID starting with 4942b4d4622045c7ded037ee01ff435c7d609e90a7a7365fd03221521c0f8d06 not found: ID does not exist" containerID="4942b4d4622045c7ded037ee01ff435c7d609e90a7a7365fd03221521c0f8d06"
Aug 05 17:36:54.821552 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.821451    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4942b4d4622045c7ded037ee01ff435c7d609e90a7a7365fd03221521c0f8d06"} err="failed to get container status \"4942b4d4622045c7ded037ee01ff435c7d609e90a7a7365fd03221521c0f8d06\": rpc error: code = NotFound desc = could not find container \"4942b4d4622045c7ded037ee01ff435c7d609e90a7a7365fd03221521c0f8d06\": container with ID starting with 4942b4d4622045c7ded037ee01ff435c7d609e90a7a7365fd03221521c0f8d06 not found: ID does not exist"
Aug 05 17:36:54.821552 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.821478    2574 scope.go:117] "RemoveContainer" containerID="b39525bdd73827a4d8ffa43e445530d9d335d6ac3265a92c808e6fbb13e39235"
Aug 05 17:36:54.821868 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:36:54.821832    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b39525bdd73827a4d8ffa43e445530d9d335d6ac3265a92c808e6fbb13e39235\": container with ID starting with b39525bdd73827a4d8ffa43e445530d9d335d6ac3265a92c808e6fbb13e39235 not found: ID does not exist" containerID="b39525bdd73827a4d8ffa43e445530d9d335d6ac3265a92c808e6fbb13e39235"
Aug 05 17:36:54.821981 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.821867    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b39525bdd73827a4d8ffa43e445530d9d335d6ac3265a92c808e6fbb13e39235"} err="failed to get container status \"b39525bdd73827a4d8ffa43e445530d9d335d6ac3265a92c808e6fbb13e39235\": rpc error: code = NotFound desc = could not find container \"b39525bdd73827a4d8ffa43e445530d9d335d6ac3265a92c808e6fbb13e39235\": container with ID starting with b39525bdd73827a4d8ffa43e445530d9d335d6ac3265a92c808e6fbb13e39235 not found: ID does not exist"
Aug 05 17:36:54.821981 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.821891    2574 scope.go:117] "RemoveContainer" containerID="5aba46e5ef91a47814d2fb4df26f79a2ea2beaefd1d872ab4ef9b9977e0e5691"
Aug 05 17:36:54.822376 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:36:54.822186    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aba46e5ef91a47814d2fb4df26f79a2ea2beaefd1d872ab4ef9b9977e0e5691\": container with ID starting with 5aba46e5ef91a47814d2fb4df26f79a2ea2beaefd1d872ab4ef9b9977e0e5691 not found: ID does not exist" containerID="5aba46e5ef91a47814d2fb4df26f79a2ea2beaefd1d872ab4ef9b9977e0e5691"
Aug 05 17:36:54.822376 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.822212    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aba46e5ef91a47814d2fb4df26f79a2ea2beaefd1d872ab4ef9b9977e0e5691"} err="failed to get container status \"5aba46e5ef91a47814d2fb4df26f79a2ea2beaefd1d872ab4ef9b9977e0e5691\": rpc error: code = NotFound desc = could not find container \"5aba46e5ef91a47814d2fb4df26f79a2ea2beaefd1d872ab4ef9b9977e0e5691\": container with ID starting with 5aba46e5ef91a47814d2fb4df26f79a2ea2beaefd1d872ab4ef9b9977e0e5691 not found: ID does not exist"
Aug 05 17:36:54.823331 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:54.823312    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7879794c64-tvsrl"]
Aug 05 17:36:55.192556 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:55.192460    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/983a04c7-ea54-4af8-bcb5-dc449a407838-proxy-tls\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:55.194860 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:55.194840    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/983a04c7-ea54-4af8-bcb5-dc449a407838-proxy-tls\") pod \"isvc-canary-promote-predictor-5cc67bffb8-m4ztq\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:55.458890 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:55.458789    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:36:55.591901 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:55.591862    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"]
Aug 05 17:36:55.594997 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:36:55.594963    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983a04c7_ea54_4af8_bcb5_dc449a407838.slice/crio-864c3684119f04432bea7f5daa3ce148793a6cd27d785c61416a615cb6821d49 WatchSource:0}: Error finding container 864c3684119f04432bea7f5daa3ce148793a6cd27d785c61416a615cb6821d49: Status 404 returned error can't find the container with id 864c3684119f04432bea7f5daa3ce148793a6cd27d785c61416a615cb6821d49
Aug 05 17:36:55.780968 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:55.780934    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" event={"ID":"983a04c7-ea54-4af8-bcb5-dc449a407838","Type":"ContainerStarted","Data":"c8b5050a86d16bdec79a83f2ebb480ba802bf899a282b5c37fefb13df1945e16"}
Aug 05 17:36:55.781166 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:55.780989    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" event={"ID":"983a04c7-ea54-4af8-bcb5-dc449a407838","Type":"ContainerStarted","Data":"864c3684119f04432bea7f5daa3ce148793a6cd27d785c61416a615cb6821d49"}
Aug 05 17:36:56.193102 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:56.193006    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5040bb41-31d4-4284-a7da-84dce23bf655" path="/var/lib/kubelet/pods/5040bb41-31d4-4284-a7da-84dce23bf655/volumes"
Aug 05 17:36:56.193526 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:56.193512    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa6ec46a-cb12-48d5-ab87-46be6b6eb481" path="/var/lib/kubelet/pods/fa6ec46a-cb12-48d5-ab87-46be6b6eb481/volumes"
Aug 05 17:36:59.796715 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:59.796683    2574 generic.go:356] "Generic (PLEG): container finished" podID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerID="c8b5050a86d16bdec79a83f2ebb480ba802bf899a282b5c37fefb13df1945e16" exitCode=0
Aug 05 17:36:59.797117 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:36:59.796757    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" event={"ID":"983a04c7-ea54-4af8-bcb5-dc449a407838","Type":"ContainerDied","Data":"c8b5050a86d16bdec79a83f2ebb480ba802bf899a282b5c37fefb13df1945e16"}
Aug 05 17:37:00.802033 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:37:00.801990    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" event={"ID":"983a04c7-ea54-4af8-bcb5-dc449a407838","Type":"ContainerStarted","Data":"de74939c3504dc9b13d195e49ffaecbcf73f82e1a09e9cfd5166897cbfbab022"}
Aug 05 17:37:00.802466 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:37:00.802067    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" event={"ID":"983a04c7-ea54-4af8-bcb5-dc449a407838","Type":"ContainerStarted","Data":"be29841c0eb5bf0c276d091b3933caa218729b5b388c8f172d511fbc209e5f58"}
Aug 05 17:37:00.802466 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:37:00.802290    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:37:00.823523 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:37:00.823471    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" podStartSLOduration=3.028676385 podStartE2EDuration="6.823455987s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.794779602s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:37:00.82239719 +0000 UTC m=+1471.114252034" watchObservedRunningTime="2026-08-05 17:37:00.823455987 +0000 UTC m=+1471.115310818"
Aug 05 17:37:01.806144 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:37:01.806107    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:37:01.807322 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:37:01.807297    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 05 17:37:02.809518 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:37:02.809480    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 05 17:37:07.814431 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:37:07.814396    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:37:07.814896 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:37:07.814867    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 05 17:37:17.815708 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:37:17.815664    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 05 17:37:27.815052 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:37:27.815013    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 05 17:37:37.815278 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:37:37.815230    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 05 17:37:47.815669 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:37:47.815621    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 05 17:37:57.815604 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:37:57.815514    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 05 17:38:07.815637 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:07.815592    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 05 17:38:17.815262 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:17.815232    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:38:24.674417 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.674378    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"]
Aug 05 17:38:24.677126 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.674660    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kserve-container" containerID="cri-o://be29841c0eb5bf0c276d091b3933caa218729b5b388c8f172d511fbc209e5f58" gracePeriod=30
Aug 05 17:38:24.677126 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.674693    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kube-rbac-proxy" containerID="cri-o://de74939c3504dc9b13d195e49ffaecbcf73f82e1a09e9cfd5166897cbfbab022" gracePeriod=30
Aug 05 17:38:24.742248 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.742213    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"]
Aug 05 17:38:24.745422 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.745400    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:24.747782 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.747763    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Aug 05 17:38:24.759134 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.759112    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4986fbaf-3d30-4b39-8078-1855fa367b30-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:24.759236 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.759145    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4986fbaf-3d30-4b39-8078-1855fa367b30-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:24.759236 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.759166    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j8sr\" (UniqueName: \"kubernetes.io/projected/4986fbaf-3d30-4b39-8078-1855fa367b30-kube-api-access-5j8sr\") pod \"isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:24.759358 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.759281    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4986fbaf-3d30-4b39-8078-1855fa367b30-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:24.759358 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.759321    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4986fbaf-3d30-4b39-8078-1855fa367b30-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:24.773106 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.773084    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"]
Aug 05 17:38:24.860664 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.860631    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4986fbaf-3d30-4b39-8078-1855fa367b30-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:24.860850 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.860686    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4986fbaf-3d30-4b39-8078-1855fa367b30-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:24.860850 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.860721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j8sr\" (UniqueName: \"kubernetes.io/projected/4986fbaf-3d30-4b39-8078-1855fa367b30-kube-api-access-5j8sr\") pod \"isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:24.860850 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.860780    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4986fbaf-3d30-4b39-8078-1855fa367b30-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:24.861192 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.861100    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4986fbaf-3d30-4b39-8078-1855fa367b30-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:24.861387 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.861366    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4986fbaf-3d30-4b39-8078-1855fa367b30-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:24.861620 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.861592    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4986fbaf-3d30-4b39-8078-1855fa367b30-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:24.862055 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.862039    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4986fbaf-3d30-4b39-8078-1855fa367b30-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:24.863236 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.863217    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4986fbaf-3d30-4b39-8078-1855fa367b30-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:24.871984 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:24.871957    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j8sr\" (UniqueName: \"kubernetes.io/projected/4986fbaf-3d30-4b39-8078-1855fa367b30-kube-api-access-5j8sr\") pod \"isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:25.055407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:25.055375    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:25.103112 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:25.103070    2574 generic.go:356] "Generic (PLEG): container finished" podID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerID="de74939c3504dc9b13d195e49ffaecbcf73f82e1a09e9cfd5166897cbfbab022" exitCode=2
Aug 05 17:38:25.103272 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:25.103147    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" event={"ID":"983a04c7-ea54-4af8-bcb5-dc449a407838","Type":"ContainerDied","Data":"de74939c3504dc9b13d195e49ffaecbcf73f82e1a09e9cfd5166897cbfbab022"}
Aug 05 17:38:25.187440 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:25.187409    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"]
Aug 05 17:38:25.190158 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:38:25.190127    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4986fbaf_3d30_4b39_8078_1855fa367b30.slice/crio-7544bcb6658480838a1087798de86c3cea4b21adaecea82ad855f344a54dfb23 WatchSource:0}: Error finding container 7544bcb6658480838a1087798de86c3cea4b21adaecea82ad855f344a54dfb23: Status 404 returned error can't find the container with id 7544bcb6658480838a1087798de86c3cea4b21adaecea82ad855f344a54dfb23
Aug 05 17:38:26.108447 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:26.108408    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" event={"ID":"4986fbaf-3d30-4b39-8078-1855fa367b30","Type":"ContainerStarted","Data":"3e5a34bb7e11600984ccace907ba1f542c7e8b8a15a1b29414aa2b2146d951f3"}
Aug 05 17:38:26.108840 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:26.108466    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" event={"ID":"4986fbaf-3d30-4b39-8078-1855fa367b30","Type":"ContainerStarted","Data":"7544bcb6658480838a1087798de86c3cea4b21adaecea82ad855f344a54dfb23"}
Aug 05 17:38:27.810712 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:27.810666    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Aug 05 17:38:27.815029 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:27.815001    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 05 17:38:29.121196 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.121157    2574 generic.go:356] "Generic (PLEG): container finished" podID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerID="3e5a34bb7e11600984ccace907ba1f542c7e8b8a15a1b29414aa2b2146d951f3" exitCode=0
Aug 05 17:38:29.121196 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.121189    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" event={"ID":"4986fbaf-3d30-4b39-8078-1855fa367b30","Type":"ContainerDied","Data":"3e5a34bb7e11600984ccace907ba1f542c7e8b8a15a1b29414aa2b2146d951f3"}
Aug 05 17:38:29.518021 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.517985    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:38:29.596250 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.596221    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvfkm\" (UniqueName: \"kubernetes.io/projected/983a04c7-ea54-4af8-bcb5-dc449a407838-kube-api-access-rvfkm\") pod \"983a04c7-ea54-4af8-bcb5-dc449a407838\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") "
Aug 05 17:38:29.596429 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.596257    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/983a04c7-ea54-4af8-bcb5-dc449a407838-cabundle-cert\") pod \"983a04c7-ea54-4af8-bcb5-dc449a407838\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") "
Aug 05 17:38:29.596429 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.596296    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/983a04c7-ea54-4af8-bcb5-dc449a407838-proxy-tls\") pod \"983a04c7-ea54-4af8-bcb5-dc449a407838\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") "
Aug 05 17:38:29.596429 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.596356    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/983a04c7-ea54-4af8-bcb5-dc449a407838-kube-rbac-proxy-sar-config\") pod \"983a04c7-ea54-4af8-bcb5-dc449a407838\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") "
Aug 05 17:38:29.596429 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.596403    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/983a04c7-ea54-4af8-bcb5-dc449a407838-kserve-provision-location\") pod \"983a04c7-ea54-4af8-bcb5-dc449a407838\" (UID: \"983a04c7-ea54-4af8-bcb5-dc449a407838\") "
Aug 05 17:38:29.596725 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.596698    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983a04c7-ea54-4af8-bcb5-dc449a407838-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "983a04c7-ea54-4af8-bcb5-dc449a407838" (UID: "983a04c7-ea54-4af8-bcb5-dc449a407838"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:38:29.596725 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.596712    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983a04c7-ea54-4af8-bcb5-dc449a407838-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "983a04c7-ea54-4af8-bcb5-dc449a407838" (UID: "983a04c7-ea54-4af8-bcb5-dc449a407838"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:38:29.596875 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.596707    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983a04c7-ea54-4af8-bcb5-dc449a407838-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "983a04c7-ea54-4af8-bcb5-dc449a407838" (UID: "983a04c7-ea54-4af8-bcb5-dc449a407838"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:38:29.598333 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.598312    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/983a04c7-ea54-4af8-bcb5-dc449a407838-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "983a04c7-ea54-4af8-bcb5-dc449a407838" (UID: "983a04c7-ea54-4af8-bcb5-dc449a407838"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:38:29.598435 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.598368    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983a04c7-ea54-4af8-bcb5-dc449a407838-kube-api-access-rvfkm" (OuterVolumeSpecName: "kube-api-access-rvfkm") pod "983a04c7-ea54-4af8-bcb5-dc449a407838" (UID: "983a04c7-ea54-4af8-bcb5-dc449a407838"). InnerVolumeSpecName "kube-api-access-rvfkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:38:29.697501 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.697407    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/983a04c7-ea54-4af8-bcb5-dc449a407838-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:38:29.697501 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.697445    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/983a04c7-ea54-4af8-bcb5-dc449a407838-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:38:29.697501 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.697455    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvfkm\" (UniqueName: \"kubernetes.io/projected/983a04c7-ea54-4af8-bcb5-dc449a407838-kube-api-access-rvfkm\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:38:29.697501 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.697464    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/983a04c7-ea54-4af8-bcb5-dc449a407838-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:38:29.697501 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:29.697472    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/983a04c7-ea54-4af8-bcb5-dc449a407838-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:38:30.127662 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.127625    2574 generic.go:356] "Generic (PLEG): container finished" podID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerID="be29841c0eb5bf0c276d091b3933caa218729b5b388c8f172d511fbc209e5f58" exitCode=0
Aug 05 17:38:30.128111 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.127692    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"
Aug 05 17:38:30.128111 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.127711    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" event={"ID":"983a04c7-ea54-4af8-bcb5-dc449a407838","Type":"ContainerDied","Data":"be29841c0eb5bf0c276d091b3933caa218729b5b388c8f172d511fbc209e5f58"}
Aug 05 17:38:30.128111 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.127786    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq" event={"ID":"983a04c7-ea54-4af8-bcb5-dc449a407838","Type":"ContainerDied","Data":"864c3684119f04432bea7f5daa3ce148793a6cd27d785c61416a615cb6821d49"}
Aug 05 17:38:30.128111 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.127817    2574 scope.go:117] "RemoveContainer" containerID="de74939c3504dc9b13d195e49ffaecbcf73f82e1a09e9cfd5166897cbfbab022"
Aug 05 17:38:30.129949 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.129928    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" event={"ID":"4986fbaf-3d30-4b39-8078-1855fa367b30","Type":"ContainerStarted","Data":"cda3e09c4a9d385bb1bc809e9bd36935e1063fc503870d21db51183f8e3ce088"}
Aug 05 17:38:30.130064 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.129984    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" event={"ID":"4986fbaf-3d30-4b39-8078-1855fa367b30","Type":"ContainerStarted","Data":"e7f6457e8278301ab4131b4c824a90671e8fabe82761252de8f468a1519c8610"}
Aug 05 17:38:30.130215 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.130197    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:30.136260 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.136245    2574 scope.go:117] "RemoveContainer" containerID="be29841c0eb5bf0c276d091b3933caa218729b5b388c8f172d511fbc209e5f58"
Aug 05 17:38:30.143819 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.143803    2574 scope.go:117] "RemoveContainer" containerID="c8b5050a86d16bdec79a83f2ebb480ba802bf899a282b5c37fefb13df1945e16"
Aug 05 17:38:30.150563 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.150549    2574 scope.go:117] "RemoveContainer" containerID="de74939c3504dc9b13d195e49ffaecbcf73f82e1a09e9cfd5166897cbfbab022"
Aug 05 17:38:30.150792 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:38:30.150772    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de74939c3504dc9b13d195e49ffaecbcf73f82e1a09e9cfd5166897cbfbab022\": container with ID starting with de74939c3504dc9b13d195e49ffaecbcf73f82e1a09e9cfd5166897cbfbab022 not found: ID does not exist" containerID="de74939c3504dc9b13d195e49ffaecbcf73f82e1a09e9cfd5166897cbfbab022"
Aug 05 17:38:30.150842 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.150800    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de74939c3504dc9b13d195e49ffaecbcf73f82e1a09e9cfd5166897cbfbab022"} err="failed to get container status \"de74939c3504dc9b13d195e49ffaecbcf73f82e1a09e9cfd5166897cbfbab022\": rpc error: code = NotFound desc = could not find container \"de74939c3504dc9b13d195e49ffaecbcf73f82e1a09e9cfd5166897cbfbab022\": container with ID starting with de74939c3504dc9b13d195e49ffaecbcf73f82e1a09e9cfd5166897cbfbab022 not found: ID does not exist"
Aug 05 17:38:30.150842 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.150818    2574 scope.go:117] "RemoveContainer" containerID="be29841c0eb5bf0c276d091b3933caa218729b5b388c8f172d511fbc209e5f58"
Aug 05 17:38:30.151051 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:38:30.151035    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be29841c0eb5bf0c276d091b3933caa218729b5b388c8f172d511fbc209e5f58\": container with ID starting with be29841c0eb5bf0c276d091b3933caa218729b5b388c8f172d511fbc209e5f58 not found: ID does not exist" containerID="be29841c0eb5bf0c276d091b3933caa218729b5b388c8f172d511fbc209e5f58"
Aug 05 17:38:30.151123 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.151055    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be29841c0eb5bf0c276d091b3933caa218729b5b388c8f172d511fbc209e5f58"} err="failed to get container status \"be29841c0eb5bf0c276d091b3933caa218729b5b388c8f172d511fbc209e5f58\": rpc error: code = NotFound desc = could not find container \"be29841c0eb5bf0c276d091b3933caa218729b5b388c8f172d511fbc209e5f58\": container with ID starting with be29841c0eb5bf0c276d091b3933caa218729b5b388c8f172d511fbc209e5f58 not found: ID does not exist"
Aug 05 17:38:30.151123 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.151068    2574 scope.go:117] "RemoveContainer" containerID="c8b5050a86d16bdec79a83f2ebb480ba802bf899a282b5c37fefb13df1945e16"
Aug 05 17:38:30.151304 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:38:30.151287    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b5050a86d16bdec79a83f2ebb480ba802bf899a282b5c37fefb13df1945e16\": container with ID starting with c8b5050a86d16bdec79a83f2ebb480ba802bf899a282b5c37fefb13df1945e16 not found: ID does not exist" containerID="c8b5050a86d16bdec79a83f2ebb480ba802bf899a282b5c37fefb13df1945e16"
Aug 05 17:38:30.151345 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.151309    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b5050a86d16bdec79a83f2ebb480ba802bf899a282b5c37fefb13df1945e16"} err="failed to get container status \"c8b5050a86d16bdec79a83f2ebb480ba802bf899a282b5c37fefb13df1945e16\": rpc error: code = NotFound desc = could not find container \"c8b5050a86d16bdec79a83f2ebb480ba802bf899a282b5c37fefb13df1945e16\": container with ID starting with c8b5050a86d16bdec79a83f2ebb480ba802bf899a282b5c37fefb13df1945e16 not found: ID does not exist"
Aug 05 17:38:30.156986 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.156944    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" podStartSLOduration=2.478785153 podStartE2EDuration="6.156932274s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.678147121s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:38:30.156371772 +0000 UTC m=+1560.448226615" watchObservedRunningTime="2026-08-05 17:38:30.156932274 +0000 UTC m=+1560.448787104"
Aug 05 17:38:30.171022 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.170996    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"]
Aug 05 17:38:30.178257 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.178235    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5cc67bffb8-m4ztq"]
Aug 05 17:38:30.194834 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:30.194801    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" path="/var/lib/kubelet/pods/983a04c7-ea54-4af8-bcb5-dc449a407838/volumes"
Aug 05 17:38:31.133967 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:31.133936    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:31.135102 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:31.135057    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 05 17:38:32.137787 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:32.137741    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 05 17:38:37.141862 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:37.141831    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:38:37.142455 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:37.142426    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 05 17:38:47.142670 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:47.142626    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 05 17:38:57.142697 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:38:57.142652    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 05 17:39:07.143260 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:07.143219    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 05 17:39:17.142685 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:17.142644    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 05 17:39:27.142741 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:27.142653    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 05 17:39:37.142770 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:37.142730    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 05 17:39:47.142915 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:47.142883    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:39:54.969841 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:54.969797    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"]
Aug 05 17:39:54.970338 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:54.970208    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kserve-container" containerID="cri-o://e7f6457e8278301ab4131b4c824a90671e8fabe82761252de8f468a1519c8610" gracePeriod=30
Aug 05 17:39:54.970592 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:54.970540    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kube-rbac-proxy" containerID="cri-o://cda3e09c4a9d385bb1bc809e9bd36935e1063fc503870d21db51183f8e3ce088" gracePeriod=30
Aug 05 17:39:55.428532 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:55.428497    2574 generic.go:356] "Generic (PLEG): container finished" podID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerID="cda3e09c4a9d385bb1bc809e9bd36935e1063fc503870d21db51183f8e3ce088" exitCode=2
Aug 05 17:39:55.428715 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:55.428592    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" event={"ID":"4986fbaf-3d30-4b39-8078-1855fa367b30","Type":"ContainerDied","Data":"cda3e09c4a9d385bb1bc809e9bd36935e1063fc503870d21db51183f8e3ce088"}
Aug 05 17:39:57.138565 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:57.138524    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Aug 05 17:39:57.142872 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:57.142846    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 05 17:39:59.919036 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:59.919014    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:39:59.974448 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:59.974359    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j8sr\" (UniqueName: \"kubernetes.io/projected/4986fbaf-3d30-4b39-8078-1855fa367b30-kube-api-access-5j8sr\") pod \"4986fbaf-3d30-4b39-8078-1855fa367b30\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") "
Aug 05 17:39:59.974448 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:59.974403    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4986fbaf-3d30-4b39-8078-1855fa367b30-kube-rbac-proxy-sar-config\") pod \"4986fbaf-3d30-4b39-8078-1855fa367b30\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") "
Aug 05 17:39:59.974448 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:59.974439    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4986fbaf-3d30-4b39-8078-1855fa367b30-proxy-tls\") pod \"4986fbaf-3d30-4b39-8078-1855fa367b30\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") "
Aug 05 17:39:59.974731 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:59.974533    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4986fbaf-3d30-4b39-8078-1855fa367b30-kserve-provision-location\") pod \"4986fbaf-3d30-4b39-8078-1855fa367b30\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") "
Aug 05 17:39:59.974731 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:59.974609    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4986fbaf-3d30-4b39-8078-1855fa367b30-cabundle-cert\") pod \"4986fbaf-3d30-4b39-8078-1855fa367b30\" (UID: \"4986fbaf-3d30-4b39-8078-1855fa367b30\") "
Aug 05 17:39:59.974861 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:59.974836    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4986fbaf-3d30-4b39-8078-1855fa367b30-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4986fbaf-3d30-4b39-8078-1855fa367b30" (UID: "4986fbaf-3d30-4b39-8078-1855fa367b30"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:39:59.974919 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:59.974895    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4986fbaf-3d30-4b39-8078-1855fa367b30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4986fbaf-3d30-4b39-8078-1855fa367b30" (UID: "4986fbaf-3d30-4b39-8078-1855fa367b30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:39:59.974968 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:59.974955    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4986fbaf-3d30-4b39-8078-1855fa367b30-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4986fbaf-3d30-4b39-8078-1855fa367b30" (UID: "4986fbaf-3d30-4b39-8078-1855fa367b30"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:39:59.976546 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:59.976516    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4986fbaf-3d30-4b39-8078-1855fa367b30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4986fbaf-3d30-4b39-8078-1855fa367b30" (UID: "4986fbaf-3d30-4b39-8078-1855fa367b30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:39:59.976546 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:39:59.976535    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4986fbaf-3d30-4b39-8078-1855fa367b30-kube-api-access-5j8sr" (OuterVolumeSpecName: "kube-api-access-5j8sr") pod "4986fbaf-3d30-4b39-8078-1855fa367b30" (UID: "4986fbaf-3d30-4b39-8078-1855fa367b30"). InnerVolumeSpecName "kube-api-access-5j8sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:40:00.075653 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.075619    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4986fbaf-3d30-4b39-8078-1855fa367b30-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:40:00.075653 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.075648    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j8sr\" (UniqueName: \"kubernetes.io/projected/4986fbaf-3d30-4b39-8078-1855fa367b30-kube-api-access-5j8sr\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:40:00.075885 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.075663    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4986fbaf-3d30-4b39-8078-1855fa367b30-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:40:00.075885 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.075677    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4986fbaf-3d30-4b39-8078-1855fa367b30-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:40:00.075885 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.075688    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4986fbaf-3d30-4b39-8078-1855fa367b30-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:40:00.449166 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.449132    2574 generic.go:356] "Generic (PLEG): container finished" podID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerID="e7f6457e8278301ab4131b4c824a90671e8fabe82761252de8f468a1519c8610" exitCode=0
Aug 05 17:40:00.449339 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.449190    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"
Aug 05 17:40:00.449339 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.449214    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" event={"ID":"4986fbaf-3d30-4b39-8078-1855fa367b30","Type":"ContainerDied","Data":"e7f6457e8278301ab4131b4c824a90671e8fabe82761252de8f468a1519c8610"}
Aug 05 17:40:00.449339 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.449286    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd" event={"ID":"4986fbaf-3d30-4b39-8078-1855fa367b30","Type":"ContainerDied","Data":"7544bcb6658480838a1087798de86c3cea4b21adaecea82ad855f344a54dfb23"}
Aug 05 17:40:00.449339 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.449326    2574 scope.go:117] "RemoveContainer" containerID="cda3e09c4a9d385bb1bc809e9bd36935e1063fc503870d21db51183f8e3ce088"
Aug 05 17:40:00.456961 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.456942    2574 scope.go:117] "RemoveContainer" containerID="e7f6457e8278301ab4131b4c824a90671e8fabe82761252de8f468a1519c8610"
Aug 05 17:40:00.465111 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.465065    2574 scope.go:117] "RemoveContainer" containerID="3e5a34bb7e11600984ccace907ba1f542c7e8b8a15a1b29414aa2b2146d951f3"
Aug 05 17:40:00.470621 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.470599    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"]
Aug 05 17:40:00.472517 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.472504    2574 scope.go:117] "RemoveContainer" containerID="cda3e09c4a9d385bb1bc809e9bd36935e1063fc503870d21db51183f8e3ce088"
Aug 05 17:40:00.472743 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:40:00.472724    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda3e09c4a9d385bb1bc809e9bd36935e1063fc503870d21db51183f8e3ce088\": container with ID starting with cda3e09c4a9d385bb1bc809e9bd36935e1063fc503870d21db51183f8e3ce088 not found: ID does not exist" containerID="cda3e09c4a9d385bb1bc809e9bd36935e1063fc503870d21db51183f8e3ce088"
Aug 05 17:40:00.472794 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.472750    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda3e09c4a9d385bb1bc809e9bd36935e1063fc503870d21db51183f8e3ce088"} err="failed to get container status \"cda3e09c4a9d385bb1bc809e9bd36935e1063fc503870d21db51183f8e3ce088\": rpc error: code = NotFound desc = could not find container \"cda3e09c4a9d385bb1bc809e9bd36935e1063fc503870d21db51183f8e3ce088\": container with ID starting with cda3e09c4a9d385bb1bc809e9bd36935e1063fc503870d21db51183f8e3ce088 not found: ID does not exist"
Aug 05 17:40:00.472794 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.472766    2574 scope.go:117] "RemoveContainer" containerID="e7f6457e8278301ab4131b4c824a90671e8fabe82761252de8f468a1519c8610"
Aug 05 17:40:00.472953 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:40:00.472937    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f6457e8278301ab4131b4c824a90671e8fabe82761252de8f468a1519c8610\": container with ID starting with e7f6457e8278301ab4131b4c824a90671e8fabe82761252de8f468a1519c8610 not found: ID does not exist" containerID="e7f6457e8278301ab4131b4c824a90671e8fabe82761252de8f468a1519c8610"
Aug 05 17:40:00.472999 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.472955    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f6457e8278301ab4131b4c824a90671e8fabe82761252de8f468a1519c8610"} err="failed to get container status \"e7f6457e8278301ab4131b4c824a90671e8fabe82761252de8f468a1519c8610\": rpc error: code = NotFound desc = could not find container \"e7f6457e8278301ab4131b4c824a90671e8fabe82761252de8f468a1519c8610\": container with ID starting with e7f6457e8278301ab4131b4c824a90671e8fabe82761252de8f468a1519c8610 not found: ID does not exist"
Aug 05 17:40:00.472999 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.472967    2574 scope.go:117] "RemoveContainer" containerID="3e5a34bb7e11600984ccace907ba1f542c7e8b8a15a1b29414aa2b2146d951f3"
Aug 05 17:40:00.473212 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:40:00.473197    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5a34bb7e11600984ccace907ba1f542c7e8b8a15a1b29414aa2b2146d951f3\": container with ID starting with 3e5a34bb7e11600984ccace907ba1f542c7e8b8a15a1b29414aa2b2146d951f3 not found: ID does not exist" containerID="3e5a34bb7e11600984ccace907ba1f542c7e8b8a15a1b29414aa2b2146d951f3"
Aug 05 17:40:00.473267 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.473214    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5a34bb7e11600984ccace907ba1f542c7e8b8a15a1b29414aa2b2146d951f3"} err="failed to get container status \"3e5a34bb7e11600984ccace907ba1f542c7e8b8a15a1b29414aa2b2146d951f3\": rpc error: code = NotFound desc = could not find container \"3e5a34bb7e11600984ccace907ba1f542c7e8b8a15a1b29414aa2b2146d951f3\": container with ID starting with 3e5a34bb7e11600984ccace907ba1f542c7e8b8a15a1b29414aa2b2146d951f3 not found: ID does not exist"
Aug 05 17:40:00.475914 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:00.475891    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7fb86b4d87-m98fd"]
Aug 05 17:40:02.192828 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:02.192792    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" path="/var/lib/kubelet/pods/4986fbaf-3d30-4b39-8078-1855fa367b30/volumes"
Aug 05 17:40:19.173882 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.173846    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"]
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174105    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="storage-initializer"
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174117    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="storage-initializer"
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174125    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kserve-container"
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174131    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kserve-container"
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174144    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kserve-container"
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174150    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kserve-container"
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174158    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="storage-initializer"
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174164    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="storage-initializer"
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174171    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kube-rbac-proxy"
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174176    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kube-rbac-proxy"
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174187    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kube-rbac-proxy"
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174193    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kube-rbac-proxy"
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174232    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kserve-container"
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174240    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kube-rbac-proxy"
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174246    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="983a04c7-ea54-4af8-bcb5-dc449a407838" containerName="kube-rbac-proxy"
Aug 05 17:40:19.174407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.174251    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4986fbaf-3d30-4b39-8078-1855fa367b30" containerName="kserve-container"
Aug 05 17:40:19.177589 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.177569    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.180769 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.180728    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Aug 05 17:40:19.180769 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.180768    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 05 17:40:19.180981 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.180818    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 05 17:40:19.180981 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.180855    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Aug 05 17:40:19.180981 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.180944    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-h9rpc\""
Aug 05 17:40:19.181831 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.181808    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 05 17:40:19.181922 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.181841    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 17:40:19.191745 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.191721    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"]
Aug 05 17:40:19.312562 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.312517    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89d0c13b-009e-4991-8781-65092ae1d872-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.312562 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.312569    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbnhg\" (UniqueName: \"kubernetes.io/projected/89d0c13b-009e-4991-8781-65092ae1d872-kube-api-access-tbnhg\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.312783 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.312643    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89d0c13b-009e-4991-8781-65092ae1d872-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.312783 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.312758    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89d0c13b-009e-4991-8781-65092ae1d872-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.312866 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.312781    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89d0c13b-009e-4991-8781-65092ae1d872-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.413904 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.413858    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89d0c13b-009e-4991-8781-65092ae1d872-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.413904 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.413909    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbnhg\" (UniqueName: \"kubernetes.io/projected/89d0c13b-009e-4991-8781-65092ae1d872-kube-api-access-tbnhg\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.414216 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.413945    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89d0c13b-009e-4991-8781-65092ae1d872-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.414216 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.413985    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89d0c13b-009e-4991-8781-65092ae1d872-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.414216 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.414011    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89d0c13b-009e-4991-8781-65092ae1d872-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.414216 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:40:19.414024    2574 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-serving-cert: secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Aug 05 17:40:19.414216 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:40:19.414118    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89d0c13b-009e-4991-8781-65092ae1d872-proxy-tls podName:89d0c13b-009e-4991-8781-65092ae1d872 nodeName:}" failed. No retries permitted until 2026-08-05 17:40:19.914097608 +0000 UTC m=+1670.205952432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/89d0c13b-009e-4991-8781-65092ae1d872-proxy-tls") pod "isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" (UID: "89d0c13b-009e-4991-8781-65092ae1d872") : secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Aug 05 17:40:19.414455 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.414407    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89d0c13b-009e-4991-8781-65092ae1d872-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.414813 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.414792    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89d0c13b-009e-4991-8781-65092ae1d872-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.414851 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.414792    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89d0c13b-009e-4991-8781-65092ae1d872-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.423959 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.423900    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbnhg\" (UniqueName: \"kubernetes.io/projected/89d0c13b-009e-4991-8781-65092ae1d872-kube-api-access-tbnhg\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.918265 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.918236    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89d0c13b-009e-4991-8781-65092ae1d872-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:19.920689 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:19.920668    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89d0c13b-009e-4991-8781-65092ae1d872-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:20.088868 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:20.088829    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:20.231153 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:20.231118    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"]
Aug 05 17:40:20.234053 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:40:20.234023    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d0c13b_009e_4991_8781_65092ae1d872.slice/crio-55e0431c6761ff892960638386642d0d07f391589e89c9e620b266f8e9d90c77 WatchSource:0}: Error finding container 55e0431c6761ff892960638386642d0d07f391589e89c9e620b266f8e9d90c77: Status 404 returned error can't find the container with id 55e0431c6761ff892960638386642d0d07f391589e89c9e620b266f8e9d90c77
Aug 05 17:40:20.518751 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:20.518711    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" event={"ID":"89d0c13b-009e-4991-8781-65092ae1d872","Type":"ContainerStarted","Data":"85dcdc584a4efc96a04f2f46298498567f8272af9230c685ca87dc12ae0edb87"}
Aug 05 17:40:20.518928 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:20.518775    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" event={"ID":"89d0c13b-009e-4991-8781-65092ae1d872","Type":"ContainerStarted","Data":"55e0431c6761ff892960638386642d0d07f391589e89c9e620b266f8e9d90c77"}
Aug 05 17:40:24.535526 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:24.535494    2574 generic.go:356] "Generic (PLEG): container finished" podID="89d0c13b-009e-4991-8781-65092ae1d872" containerID="85dcdc584a4efc96a04f2f46298498567f8272af9230c685ca87dc12ae0edb87" exitCode=0
Aug 05 17:40:24.536028 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:24.535562    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" event={"ID":"89d0c13b-009e-4991-8781-65092ae1d872","Type":"ContainerDied","Data":"85dcdc584a4efc96a04f2f46298498567f8272af9230c685ca87dc12ae0edb87"}
Aug 05 17:40:25.542163 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:25.542124    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" event={"ID":"89d0c13b-009e-4991-8781-65092ae1d872","Type":"ContainerStarted","Data":"ed54b09b05737acb7cbf0714a83512f644530de02143cba27d3004ea208fbdd3"}
Aug 05 17:40:25.542658 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:25.542208    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" event={"ID":"89d0c13b-009e-4991-8781-65092ae1d872","Type":"ContainerStarted","Data":"c90d1c6186390355b892ccc470c7418446f552c83f0d0021de2851b6ca9764db"}
Aug 05 17:40:25.542658 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:25.542244    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" event={"ID":"89d0c13b-009e-4991-8781-65092ae1d872","Type":"ContainerStarted","Data":"f9224760adcd32a12e644ba81b63adeef2c114447551a4efe924cfbd0520ea43"}
Aug 05 17:40:25.542658 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:25.542496    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:25.568908 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:25.568857    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podStartSLOduration=2.815499547 podStartE2EDuration="6.568840289s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.753340742s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:40:25.566940696 +0000 UTC m=+1675.858795524" watchObservedRunningTime="2026-08-05 17:40:25.568840289 +0000 UTC m=+1675.860695120"
Aug 05 17:40:26.546768 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:26.546740    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:26.547159 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:26.546796    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:26.548147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:26.548116    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 05 17:40:26.548791 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:26.548764    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:40:27.550354 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:27.550307    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 05 17:40:27.550758 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:27.550595    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:40:32.555208 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:32.555180    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:40:32.555846 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:32.555820    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 05 17:40:32.556007 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:32.555983    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:40:42.555929 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:42.555885    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 05 17:40:42.556409 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:42.556381    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:40:52.556515 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:52.556423    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 05 17:40:52.557104 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:40:52.556942    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:41:02.555823 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:02.555774    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 05 17:41:02.556338 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:02.556226    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:41:12.556636 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:12.556594    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 05 17:41:12.557129 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:12.557100    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:41:22.555853 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:22.555810    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 05 17:41:22.556337 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:22.556306    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:41:32.555849 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:32.555801    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 05 17:41:32.556439 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:32.556240    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:41:42.556169 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:42.556135    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:41:42.556559 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:42.556227    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:41:54.483420 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:54.483380    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"]
Aug 05 17:41:54.483883 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:54.483633    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container" containerID="cri-o://f9224760adcd32a12e644ba81b63adeef2c114447551a4efe924cfbd0520ea43" gracePeriod=30
Aug 05 17:41:54.483883 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:54.483671    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kube-rbac-proxy" containerID="cri-o://c90d1c6186390355b892ccc470c7418446f552c83f0d0021de2851b6ca9764db" gracePeriod=30
Aug 05 17:41:54.483883 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:54.483664    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent" containerID="cri-o://ed54b09b05737acb7cbf0714a83512f644530de02143cba27d3004ea208fbdd3" gracePeriod=30
Aug 05 17:41:54.867645 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:54.867610    2574 generic.go:356] "Generic (PLEG): container finished" podID="89d0c13b-009e-4991-8781-65092ae1d872" containerID="c90d1c6186390355b892ccc470c7418446f552c83f0d0021de2851b6ca9764db" exitCode=2
Aug 05 17:41:54.867818 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:54.867683    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" event={"ID":"89d0c13b-009e-4991-8781-65092ae1d872","Type":"ContainerDied","Data":"c90d1c6186390355b892ccc470c7418446f552c83f0d0021de2851b6ca9764db"}
Aug 05 17:41:57.550632 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:57.550584    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Aug 05 17:41:59.887991 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:59.887958    2574 generic.go:356] "Generic (PLEG): container finished" podID="89d0c13b-009e-4991-8781-65092ae1d872" containerID="f9224760adcd32a12e644ba81b63adeef2c114447551a4efe924cfbd0520ea43" exitCode=0
Aug 05 17:41:59.888384 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:41:59.888011    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" event={"ID":"89d0c13b-009e-4991-8781-65092ae1d872","Type":"ContainerDied","Data":"f9224760adcd32a12e644ba81b63adeef2c114447551a4efe924cfbd0520ea43"}
Aug 05 17:42:02.551363 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:02.551320    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Aug 05 17:42:02.555689 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:02.555657    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 05 17:42:02.555981 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:02.555958    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:42:07.550456 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:07.550414    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Aug 05 17:42:07.550952 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:07.550567    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:42:11.585304 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:11.585272    2574 ???:1] "http: TLS handshake error from 10.0.142.192:36770: EOF"
Aug 05 17:42:11.588325 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:11.588297    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kserve-container/0.log"
Aug 05 17:42:11.614487 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:11.614455    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kube-rbac-proxy/0.log"
Aug 05 17:42:11.628387 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:11.628359    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/agent/0.log"
Aug 05 17:42:11.644039 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:11.644014    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/storage-initializer/0.log"
Aug 05 17:42:12.473629 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:12.473599    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kserve-container/0.log"
Aug 05 17:42:12.488503 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:12.488459    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kube-rbac-proxy/0.log"
Aug 05 17:42:12.501649 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:12.501618    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/agent/0.log"
Aug 05 17:42:12.515100 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:12.515062    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/storage-initializer/0.log"
Aug 05 17:42:12.551431 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:12.551388    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Aug 05 17:42:12.555731 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:12.555704    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 05 17:42:12.556030 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:12.556005    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:42:13.423686 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:13.423651    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kserve-container/0.log"
Aug 05 17:42:13.438384 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:13.438354    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kube-rbac-proxy/0.log"
Aug 05 17:42:13.451244 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:13.451220    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/agent/0.log"
Aug 05 17:42:13.467189 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:13.467166    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/storage-initializer/0.log"
Aug 05 17:42:14.291070 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:14.291038    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kserve-container/0.log"
Aug 05 17:42:14.307661 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:14.307635    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kube-rbac-proxy/0.log"
Aug 05 17:42:14.320666 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:14.320634    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/agent/0.log"
Aug 05 17:42:14.333241 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:14.333222    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/storage-initializer/0.log"
Aug 05 17:42:15.158147 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:15.158118    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kserve-container/0.log"
Aug 05 17:42:15.203256 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:15.203228    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kube-rbac-proxy/0.log"
Aug 05 17:42:15.218461 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:15.218434    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/agent/0.log"
Aug 05 17:42:15.231799 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:15.231772    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/storage-initializer/0.log"
Aug 05 17:42:16.070255 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:16.070201    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kserve-container/0.log"
Aug 05 17:42:16.086948 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:16.086924    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kube-rbac-proxy/0.log"
Aug 05 17:42:16.102722 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:16.102700    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/agent/0.log"
Aug 05 17:42:16.116530 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:16.116482    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/storage-initializer/0.log"
Aug 05 17:42:16.930918 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:16.930875    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kserve-container/0.log"
Aug 05 17:42:16.949089 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:16.949048    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kube-rbac-proxy/0.log"
Aug 05 17:42:16.962620 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:16.962590    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/agent/0.log"
Aug 05 17:42:16.977243 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:16.977216    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/storage-initializer/0.log"
Aug 05 17:42:17.550516 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:17.550467    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Aug 05 17:42:17.852097 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:17.851990    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kserve-container/0.log"
Aug 05 17:42:17.867747 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:17.867722    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kube-rbac-proxy/0.log"
Aug 05 17:42:17.880704 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:17.880679    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/agent/0.log"
Aug 05 17:42:17.901436 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:17.901405    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/storage-initializer/0.log"
Aug 05 17:42:18.812328 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:18.812299    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kserve-container/0.log"
Aug 05 17:42:18.829025 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:18.828996    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kube-rbac-proxy/0.log"
Aug 05 17:42:18.842845 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:18.842816    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/agent/0.log"
Aug 05 17:42:18.859734 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:18.859711    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/storage-initializer/0.log"
Aug 05 17:42:19.697334 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:19.697299    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kserve-container/0.log"
Aug 05 17:42:19.715374 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:19.715349    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kube-rbac-proxy/0.log"
Aug 05 17:42:19.734663 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:19.734638    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/agent/0.log"
Aug 05 17:42:19.760093 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:19.760049    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/storage-initializer/0.log"
Aug 05 17:42:20.623700 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:20.623671    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kserve-container/0.log"
Aug 05 17:42:20.642465 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:20.642428    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kube-rbac-proxy/0.log"
Aug 05 17:42:20.655355 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:20.655333    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/agent/0.log"
Aug 05 17:42:20.669127 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:20.669101    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/storage-initializer/0.log"
Aug 05 17:42:21.505610 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:21.505576    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kserve-container/0.log"
Aug 05 17:42:21.524388 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:21.524360    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kube-rbac-proxy/0.log"
Aug 05 17:42:21.538309 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:21.538288    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/agent/0.log"
Aug 05 17:42:21.554033 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:21.554002    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/storage-initializer/0.log"
Aug 05 17:42:22.400095 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:22.400054    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kserve-container/0.log"
Aug 05 17:42:22.420484 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:22.420457    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kube-rbac-proxy/0.log"
Aug 05 17:42:22.436508 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:22.436480    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/agent/0.log"
Aug 05 17:42:22.450342 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:22.450317    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/storage-initializer/0.log"
Aug 05 17:42:22.550443 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:22.550401    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Aug 05 17:42:22.555764 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:22.555735    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Aug 05 17:42:22.555920 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:22.555860    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:42:22.556130 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:22.556107    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:42:22.556206 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:22.556195    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:42:23.513427 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:23.513396    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kserve-container/0.log"
Aug 05 17:42:23.529585 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:23.529560    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/kube-rbac-proxy/0.log"
Aug 05 17:42:23.548977 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:23.548950    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/agent/0.log"
Aug 05 17:42:23.566499 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:23.566473    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5_89d0c13b-009e-4991-8781-65092ae1d872/storage-initializer/0.log"
Aug 05 17:42:24.674680 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.674655    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:42:24.833518 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.833479    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89d0c13b-009e-4991-8781-65092ae1d872-kube-rbac-proxy-sar-config\") pod \"89d0c13b-009e-4991-8781-65092ae1d872\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") "
Aug 05 17:42:24.833518 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.833523    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbnhg\" (UniqueName: \"kubernetes.io/projected/89d0c13b-009e-4991-8781-65092ae1d872-kube-api-access-tbnhg\") pod \"89d0c13b-009e-4991-8781-65092ae1d872\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") "
Aug 05 17:42:24.833744 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.833546    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89d0c13b-009e-4991-8781-65092ae1d872-proxy-tls\") pod \"89d0c13b-009e-4991-8781-65092ae1d872\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") "
Aug 05 17:42:24.833744 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.833593    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89d0c13b-009e-4991-8781-65092ae1d872-kserve-provision-location\") pod \"89d0c13b-009e-4991-8781-65092ae1d872\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") "
Aug 05 17:42:24.833744 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.833623    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89d0c13b-009e-4991-8781-65092ae1d872-cabundle-cert\") pod \"89d0c13b-009e-4991-8781-65092ae1d872\" (UID: \"89d0c13b-009e-4991-8781-65092ae1d872\") "
Aug 05 17:42:24.833976 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.833944    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d0c13b-009e-4991-8781-65092ae1d872-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89d0c13b-009e-4991-8781-65092ae1d872" (UID: "89d0c13b-009e-4991-8781-65092ae1d872"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:24.834049 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.833991    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d0c13b-009e-4991-8781-65092ae1d872-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89d0c13b-009e-4991-8781-65092ae1d872" (UID: "89d0c13b-009e-4991-8781-65092ae1d872"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:42:24.834049 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.833997    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d0c13b-009e-4991-8781-65092ae1d872-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89d0c13b-009e-4991-8781-65092ae1d872" (UID: "89d0c13b-009e-4991-8781-65092ae1d872"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:42:24.835737 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.835701    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89d0c13b-009e-4991-8781-65092ae1d872-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89d0c13b-009e-4991-8781-65092ae1d872" (UID: "89d0c13b-009e-4991-8781-65092ae1d872"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:42:24.835855 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.835736    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d0c13b-009e-4991-8781-65092ae1d872-kube-api-access-tbnhg" (OuterVolumeSpecName: "kube-api-access-tbnhg") pod "89d0c13b-009e-4991-8781-65092ae1d872" (UID: "89d0c13b-009e-4991-8781-65092ae1d872"). InnerVolumeSpecName "kube-api-access-tbnhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:42:24.934059 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.934026    2574 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89d0c13b-009e-4991-8781-65092ae1d872-proxy-tls\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:42:24.934059 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.934052    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89d0c13b-009e-4991-8781-65092ae1d872-kserve-provision-location\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:42:24.934059 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.934063    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89d0c13b-009e-4991-8781-65092ae1d872-cabundle-cert\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:42:24.934318 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.934093    2574 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89d0c13b-009e-4991-8781-65092ae1d872-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:42:24.934318 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.934102    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbnhg\" (UniqueName: \"kubernetes.io/projected/89d0c13b-009e-4991-8781-65092ae1d872-kube-api-access-tbnhg\") on node \"ip-10-0-142-192.ec2.internal\" DevicePath \"\""
Aug 05 17:42:24.978712 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.978668    2574 generic.go:356] "Generic (PLEG): container finished" podID="89d0c13b-009e-4991-8781-65092ae1d872" containerID="ed54b09b05737acb7cbf0714a83512f644530de02143cba27d3004ea208fbdd3" exitCode=0
Aug 05 17:42:24.978914 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.978735    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"
Aug 05 17:42:24.978914 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.978752    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" event={"ID":"89d0c13b-009e-4991-8781-65092ae1d872","Type":"ContainerDied","Data":"ed54b09b05737acb7cbf0714a83512f644530de02143cba27d3004ea208fbdd3"}
Aug 05 17:42:24.978914 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.978816    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5" event={"ID":"89d0c13b-009e-4991-8781-65092ae1d872","Type":"ContainerDied","Data":"55e0431c6761ff892960638386642d0d07f391589e89c9e620b266f8e9d90c77"}
Aug 05 17:42:24.978914 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.978848    2574 scope.go:117] "RemoveContainer" containerID="ed54b09b05737acb7cbf0714a83512f644530de02143cba27d3004ea208fbdd3"
Aug 05 17:42:24.987557 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.987252    2574 scope.go:117] "RemoveContainer" containerID="c90d1c6186390355b892ccc470c7418446f552c83f0d0021de2851b6ca9764db"
Aug 05 17:42:24.994391 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:24.994375    2574 scope.go:117] "RemoveContainer" containerID="f9224760adcd32a12e644ba81b63adeef2c114447551a4efe924cfbd0520ea43"
Aug 05 17:42:25.001021 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:25.001003    2574 scope.go:117] "RemoveContainer" containerID="85dcdc584a4efc96a04f2f46298498567f8272af9230c685ca87dc12ae0edb87"
Aug 05 17:42:25.005516 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:25.005491    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"]
Aug 05 17:42:25.009932 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:25.009911    2574 scope.go:117] "RemoveContainer" containerID="ed54b09b05737acb7cbf0714a83512f644530de02143cba27d3004ea208fbdd3"
Aug 05 17:42:25.010230 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:42:25.010210    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed54b09b05737acb7cbf0714a83512f644530de02143cba27d3004ea208fbdd3\": container with ID starting with ed54b09b05737acb7cbf0714a83512f644530de02143cba27d3004ea208fbdd3 not found: ID does not exist" containerID="ed54b09b05737acb7cbf0714a83512f644530de02143cba27d3004ea208fbdd3"
Aug 05 17:42:25.010298 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:25.010239    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed54b09b05737acb7cbf0714a83512f644530de02143cba27d3004ea208fbdd3"} err="failed to get container status \"ed54b09b05737acb7cbf0714a83512f644530de02143cba27d3004ea208fbdd3\": rpc error: code = NotFound desc = could not find container \"ed54b09b05737acb7cbf0714a83512f644530de02143cba27d3004ea208fbdd3\": container with ID starting with ed54b09b05737acb7cbf0714a83512f644530de02143cba27d3004ea208fbdd3 not found: ID does not exist"
Aug 05 17:42:25.010298 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:25.010258    2574 scope.go:117] "RemoveContainer" containerID="c90d1c6186390355b892ccc470c7418446f552c83f0d0021de2851b6ca9764db"
Aug 05 17:42:25.010502 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:42:25.010485    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90d1c6186390355b892ccc470c7418446f552c83f0d0021de2851b6ca9764db\": container with ID starting with c90d1c6186390355b892ccc470c7418446f552c83f0d0021de2851b6ca9764db not found: ID does not exist" containerID="c90d1c6186390355b892ccc470c7418446f552c83f0d0021de2851b6ca9764db"
Aug 05 17:42:25.010540 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:25.010512    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90d1c6186390355b892ccc470c7418446f552c83f0d0021de2851b6ca9764db"} err="failed to get container status \"c90d1c6186390355b892ccc470c7418446f552c83f0d0021de2851b6ca9764db\": rpc error: code = NotFound desc = could not find container \"c90d1c6186390355b892ccc470c7418446f552c83f0d0021de2851b6ca9764db\": container with ID starting with c90d1c6186390355b892ccc470c7418446f552c83f0d0021de2851b6ca9764db not found: ID does not exist"
Aug 05 17:42:25.010540 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:25.010528    2574 scope.go:117] "RemoveContainer" containerID="f9224760adcd32a12e644ba81b63adeef2c114447551a4efe924cfbd0520ea43"
Aug 05 17:42:25.010759 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:42:25.010742    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9224760adcd32a12e644ba81b63adeef2c114447551a4efe924cfbd0520ea43\": container with ID starting with f9224760adcd32a12e644ba81b63adeef2c114447551a4efe924cfbd0520ea43 not found: ID does not exist" containerID="f9224760adcd32a12e644ba81b63adeef2c114447551a4efe924cfbd0520ea43"
Aug 05 17:42:25.010802 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:25.010764    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9224760adcd32a12e644ba81b63adeef2c114447551a4efe924cfbd0520ea43"} err="failed to get container status \"f9224760adcd32a12e644ba81b63adeef2c114447551a4efe924cfbd0520ea43\": rpc error: code = NotFound desc = could not find container \"f9224760adcd32a12e644ba81b63adeef2c114447551a4efe924cfbd0520ea43\": container with ID starting with f9224760adcd32a12e644ba81b63adeef2c114447551a4efe924cfbd0520ea43 not found: ID does not exist"
Aug 05 17:42:25.010802 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:25.010778    2574 scope.go:117] "RemoveContainer" containerID="85dcdc584a4efc96a04f2f46298498567f8272af9230c685ca87dc12ae0edb87"
Aug 05 17:42:25.011021 ip-10-0-142-192 kubenswrapper[2574]: E0805 17:42:25.011004    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85dcdc584a4efc96a04f2f46298498567f8272af9230c685ca87dc12ae0edb87\": container with ID starting with 85dcdc584a4efc96a04f2f46298498567f8272af9230c685ca87dc12ae0edb87 not found: ID does not exist" containerID="85dcdc584a4efc96a04f2f46298498567f8272af9230c685ca87dc12ae0edb87"
Aug 05 17:42:25.011063 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:25.011026    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85dcdc584a4efc96a04f2f46298498567f8272af9230c685ca87dc12ae0edb87"} err="failed to get container status \"85dcdc584a4efc96a04f2f46298498567f8272af9230c685ca87dc12ae0edb87\": rpc error: code = NotFound desc = could not find container \"85dcdc584a4efc96a04f2f46298498567f8272af9230c685ca87dc12ae0edb87\": container with ID starting with 85dcdc584a4efc96a04f2f46298498567f8272af9230c685ca87dc12ae0edb87 not found: ID does not exist"
Aug 05 17:42:25.014548 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:25.014527    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7d4b6c5f84-qmtr5"]
Aug 05 17:42:26.192927 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:26.192893    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89d0c13b-009e-4991-8781-65092ae1d872" path="/var/lib/kubelet/pods/89d0c13b-009e-4991-8781-65092ae1d872/volumes"
Aug 05 17:42:27.375407 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.375371    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jjq9m/must-gather-wb74d"]
Aug 05 17:42:27.375797 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.375684    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="storage-initializer"
Aug 05 17:42:27.375797 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.375697    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="storage-initializer"
Aug 05 17:42:27.375797 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.375708    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container"
Aug 05 17:42:27.375797 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.375714    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container"
Aug 05 17:42:27.375797 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.375743    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kube-rbac-proxy"
Aug 05 17:42:27.375797 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.375750    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kube-rbac-proxy"
Aug 05 17:42:27.375797 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.375757    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent"
Aug 05 17:42:27.375797 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.375762    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent"
Aug 05 17:42:27.375797 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.375799    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kserve-container"
Aug 05 17:42:27.376091 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.375809    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="agent"
Aug 05 17:42:27.376091 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.375819    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89d0c13b-009e-4991-8781-65092ae1d872" containerName="kube-rbac-proxy"
Aug 05 17:42:27.380516 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.380499    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jjq9m/must-gather-wb74d"
Aug 05 17:42:27.383116 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.383086    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jjq9m\"/\"kube-root-ca.crt\""
Aug 05 17:42:27.383961 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.383945    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jjq9m\"/\"default-dockercfg-fndrf\""
Aug 05 17:42:27.384033 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.383964    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jjq9m\"/\"openshift-service-ca.crt\""
Aug 05 17:42:27.387711 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.387689    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jjq9m/must-gather-wb74d"]
Aug 05 17:42:27.554826 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.554791    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3b3b2626-ca0d-4ed5-b788-11e20152180a-must-gather-output\") pod \"must-gather-wb74d\" (UID: \"3b3b2626-ca0d-4ed5-b788-11e20152180a\") " pod="openshift-must-gather-jjq9m/must-gather-wb74d"
Aug 05 17:42:27.554826 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.554825    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9f5l\" (UniqueName: \"kubernetes.io/projected/3b3b2626-ca0d-4ed5-b788-11e20152180a-kube-api-access-n9f5l\") pod \"must-gather-wb74d\" (UID: \"3b3b2626-ca0d-4ed5-b788-11e20152180a\") " pod="openshift-must-gather-jjq9m/must-gather-wb74d"
Aug 05 17:42:27.656158 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.656053    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3b3b2626-ca0d-4ed5-b788-11e20152180a-must-gather-output\") pod \"must-gather-wb74d\" (UID: \"3b3b2626-ca0d-4ed5-b788-11e20152180a\") " pod="openshift-must-gather-jjq9m/must-gather-wb74d"
Aug 05 17:42:27.656158 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.656126    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9f5l\" (UniqueName: \"kubernetes.io/projected/3b3b2626-ca0d-4ed5-b788-11e20152180a-kube-api-access-n9f5l\") pod \"must-gather-wb74d\" (UID: \"3b3b2626-ca0d-4ed5-b788-11e20152180a\") " pod="openshift-must-gather-jjq9m/must-gather-wb74d"
Aug 05 17:42:27.656399 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.656377    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3b3b2626-ca0d-4ed5-b788-11e20152180a-must-gather-output\") pod \"must-gather-wb74d\" (UID: \"3b3b2626-ca0d-4ed5-b788-11e20152180a\") " pod="openshift-must-gather-jjq9m/must-gather-wb74d"
Aug 05 17:42:27.668081 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.668056    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9f5l\" (UniqueName: \"kubernetes.io/projected/3b3b2626-ca0d-4ed5-b788-11e20152180a-kube-api-access-n9f5l\") pod \"must-gather-wb74d\" (UID: \"3b3b2626-ca0d-4ed5-b788-11e20152180a\") " pod="openshift-must-gather-jjq9m/must-gather-wb74d"
Aug 05 17:42:27.689996 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.689968    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jjq9m/must-gather-wb74d"
Aug 05 17:42:27.815870 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.815765    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jjq9m/must-gather-wb74d"]
Aug 05 17:42:27.818448 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:42:27.818420    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b3b2626_ca0d_4ed5_b788_11e20152180a.slice/crio-96f9834ec6e5e20971e6b1a8714b5de797fc8e4fd332101982f706c6dc043c53 WatchSource:0}: Error finding container 96f9834ec6e5e20971e6b1a8714b5de797fc8e4fd332101982f706c6dc043c53: Status 404 returned error can't find the container with id 96f9834ec6e5e20971e6b1a8714b5de797fc8e4fd332101982f706c6dc043c53
Aug 05 17:42:27.825683 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.825663    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 05 17:42:27.991239 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:27.991146    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjq9m/must-gather-wb74d" event={"ID":"3b3b2626-ca0d-4ed5-b788-11e20152180a","Type":"ContainerStarted","Data":"96f9834ec6e5e20971e6b1a8714b5de797fc8e4fd332101982f706c6dc043c53"}
Aug 05 17:42:28.999146 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:28.999067    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjq9m/must-gather-wb74d" event={"ID":"3b3b2626-ca0d-4ed5-b788-11e20152180a","Type":"ContainerStarted","Data":"98616fc4375509966ebb4ff263e42f1fafdd0914a6428d8989804093db9de55b"}
Aug 05 17:42:29.000333 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:29.000286    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjq9m/must-gather-wb74d" event={"ID":"3b3b2626-ca0d-4ed5-b788-11e20152180a","Type":"ContainerStarted","Data":"8afe357269f321f32c3682f20e15467064a88e7b5c4da39ecdd447f0deb2c98a"}
Aug 05 17:42:29.020538 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:29.020471    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jjq9m/must-gather-wb74d" podStartSLOduration=1.24010646 podStartE2EDuration="2.020450756s" totalImagesPullingTime="780.344296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:42:29.019140472 +0000 UTC m=+1799.310995306" watchObservedRunningTime="2026-08-05 17:42:29.020450756 +0000 UTC m=+1799.312305588"
Aug 05 17:42:30.252170 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:30.252141    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w89xx_057a0fc7-bbc2-47e0-9b9f-d4e292b354f1/global-pull-secret-syncer/0.log"
Aug 05 17:42:30.468768 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:30.468726    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wk6ks_f1ee110b-ab37-4c60-b72c-74f07b55668d/konnectivity-agent/0.log"
Aug 05 17:42:30.561526 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:30.561491    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-192.ec2.internal_322b4eab99b8f4f207438c9174b617a8/haproxy/0.log"
Aug 05 17:42:35.011484 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:35.011451    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bw488_3eb12115-1963-4b6e-bcb5-02c3fe687086/node-exporter/0.log"
Aug 05 17:42:35.047433 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:35.047402    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bw488_3eb12115-1963-4b6e-bcb5-02c3fe687086/kube-rbac-proxy/0.log"
Aug 05 17:42:35.080135 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:35.080106    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bw488_3eb12115-1963-4b6e-bcb5-02c3fe687086/init-textfile/0.log"
Aug 05 17:42:38.005466 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.005424    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"]
Aug 05 17:42:38.009009 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.008982    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.024114 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.024084    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"]
Aug 05 17:42:38.145553 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.145505    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f52998c5-0ff4-415e-a3de-9b4688d6c0ce-podres\") pod \"perf-node-gather-daemonset-pp9hj\" (UID: \"f52998c5-0ff4-415e-a3de-9b4688d6c0ce\") " pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.145553 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.145551    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f52998c5-0ff4-415e-a3de-9b4688d6c0ce-sys\") pod \"perf-node-gather-daemonset-pp9hj\" (UID: \"f52998c5-0ff4-415e-a3de-9b4688d6c0ce\") " pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.145821 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.145570    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqms5\" (UniqueName: \"kubernetes.io/projected/f52998c5-0ff4-415e-a3de-9b4688d6c0ce-kube-api-access-cqms5\") pod \"perf-node-gather-daemonset-pp9hj\" (UID: \"f52998c5-0ff4-415e-a3de-9b4688d6c0ce\") " pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.145821 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.145588    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f52998c5-0ff4-415e-a3de-9b4688d6c0ce-proc\") pod \"perf-node-gather-daemonset-pp9hj\" (UID: \"f52998c5-0ff4-415e-a3de-9b4688d6c0ce\") " pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.145821 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.145604    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f52998c5-0ff4-415e-a3de-9b4688d6c0ce-lib-modules\") pod \"perf-node-gather-daemonset-pp9hj\" (UID: \"f52998c5-0ff4-415e-a3de-9b4688d6c0ce\") " pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.246898 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.246864    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f52998c5-0ff4-415e-a3de-9b4688d6c0ce-podres\") pod \"perf-node-gather-daemonset-pp9hj\" (UID: \"f52998c5-0ff4-415e-a3de-9b4688d6c0ce\") " pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.247116 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.246904    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f52998c5-0ff4-415e-a3de-9b4688d6c0ce-sys\") pod \"perf-node-gather-daemonset-pp9hj\" (UID: \"f52998c5-0ff4-415e-a3de-9b4688d6c0ce\") " pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.247116 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.246924    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqms5\" (UniqueName: \"kubernetes.io/projected/f52998c5-0ff4-415e-a3de-9b4688d6c0ce-kube-api-access-cqms5\") pod \"perf-node-gather-daemonset-pp9hj\" (UID: \"f52998c5-0ff4-415e-a3de-9b4688d6c0ce\") " pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.247116 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.246942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f52998c5-0ff4-415e-a3de-9b4688d6c0ce-proc\") pod \"perf-node-gather-daemonset-pp9hj\" (UID: \"f52998c5-0ff4-415e-a3de-9b4688d6c0ce\") " pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.247116 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.246960    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f52998c5-0ff4-415e-a3de-9b4688d6c0ce-lib-modules\") pod \"perf-node-gather-daemonset-pp9hj\" (UID: \"f52998c5-0ff4-415e-a3de-9b4688d6c0ce\") " pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.247116 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.246996    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f52998c5-0ff4-415e-a3de-9b4688d6c0ce-sys\") pod \"perf-node-gather-daemonset-pp9hj\" (UID: \"f52998c5-0ff4-415e-a3de-9b4688d6c0ce\") " pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.247116 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.247049    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f52998c5-0ff4-415e-a3de-9b4688d6c0ce-proc\") pod \"perf-node-gather-daemonset-pp9hj\" (UID: \"f52998c5-0ff4-415e-a3de-9b4688d6c0ce\") " pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.247116 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.247107    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f52998c5-0ff4-415e-a3de-9b4688d6c0ce-podres\") pod \"perf-node-gather-daemonset-pp9hj\" (UID: \"f52998c5-0ff4-415e-a3de-9b4688d6c0ce\") " pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.247509 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.247107    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f52998c5-0ff4-415e-a3de-9b4688d6c0ce-lib-modules\") pod \"perf-node-gather-daemonset-pp9hj\" (UID: \"f52998c5-0ff4-415e-a3de-9b4688d6c0ce\") " pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.256922 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.256581    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqms5\" (UniqueName: \"kubernetes.io/projected/f52998c5-0ff4-415e-a3de-9b4688d6c0ce-kube-api-access-cqms5\") pod \"perf-node-gather-daemonset-pp9hj\" (UID: \"f52998c5-0ff4-415e-a3de-9b4688d6c0ce\") " pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.320113 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.320060    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:38.487029 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:38.486996    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"]
Aug 05 17:42:38.493060 ip-10-0-142-192 kubenswrapper[2574]: W0805 17:42:38.493009    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf52998c5_0ff4_415e_a3de_9b4688d6c0ce.slice/crio-5ea02527c5d23a6f9d489b95d40e94930b8504da61512b8fa9197fe8db90a389 WatchSource:0}: Error finding container 5ea02527c5d23a6f9d489b95d40e94930b8504da61512b8fa9197fe8db90a389: Status 404 returned error can't find the container with id 5ea02527c5d23a6f9d489b95d40e94930b8504da61512b8fa9197fe8db90a389
Aug 05 17:42:39.049111 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:39.049054    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj" event={"ID":"f52998c5-0ff4-415e-a3de-9b4688d6c0ce","Type":"ContainerStarted","Data":"2b1d3b9168f5835f2fc84d6de186947370ec964e4b0ad7ce2948c1305e3746a1"}
Aug 05 17:42:39.049532 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:39.049176    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj" event={"ID":"f52998c5-0ff4-415e-a3de-9b4688d6c0ce","Type":"ContainerStarted","Data":"5ea02527c5d23a6f9d489b95d40e94930b8504da61512b8fa9197fe8db90a389"}
Aug 05 17:42:39.049532 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:39.049239    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:39.072790 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:39.072731    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj" podStartSLOduration=2.072710446 podStartE2EDuration="2.072710446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:42:39.071190139 +0000 UTC m=+1809.363044970" watchObservedRunningTime="2026-08-05 17:42:39.072710446 +0000 UTC m=+1809.364565278"
Aug 05 17:42:40.042104 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:40.042052    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mj85f_066a1a18-72fc-4047-a6c8-527dcbca50c0/dns/0.log"
Aug 05 17:42:40.086274 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:40.086244    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mj85f_066a1a18-72fc-4047-a6c8-527dcbca50c0/kube-rbac-proxy/0.log"
Aug 05 17:42:40.237016 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:40.236982    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kgdd2_fc979956-0bb5-4600-81bb-19a5521be56d/dns-node-resolver/0.log"
Aug 05 17:42:40.901619 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:40.901585    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hsn68_ef418180-36b0-450a-8ac5-8518f605f717/node-ca/0.log"
Aug 05 17:42:42.133684 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:42.133649    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4jkwc_8feeab71-5d14-4b7b-a0d5-54e9439b5078/serve-healthcheck-canary/0.log"
Aug 05 17:42:42.765986 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:42.765956    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5ldls_f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7/kube-rbac-proxy/0.log"
Aug 05 17:42:42.791724 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:42.791699    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5ldls_f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7/exporter/0.log"
Aug 05 17:42:42.817948 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:42.817923    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5ldls_f5d5390c-bc05-4dd2-8c64-dd73ad70b7d7/extractor/0.log"
Aug 05 17:42:45.062738 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:45.062709    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jjq9m/perf-node-gather-daemonset-pp9hj"
Aug 05 17:42:45.325474 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:45.325395    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6f49dcd9ff-bvr5x_16a4a7de-e6a6-4260-a8a2-79a956677d62/manager/0.log"
Aug 05 17:42:45.604413 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:45.604332    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-8588677cfb-z6fjd_cc73af62-599e-42ec-aecd-51377ff21ae6/manager/0.log"
Aug 05 17:42:45.652455 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:45.652426    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9snm8_a70bcb32-2274-4161-966f-51520adc4c91/s3-init/0.log"
Aug 05 17:42:45.672990 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:45.672955    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9snm8_a70bcb32-2274-4161-966f-51520adc4c91/download-hf-model/0.log"
Aug 05 17:42:45.705830 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:45.705802    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-2b982_7ce1cc18-c8e3-40a3-a052-77442f8ebfe2/seaweedfs/0.log"
Aug 05 17:42:51.766304 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:51.766277    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2slm7_2347bfcd-6da3-456d-a788-a18eaf3dbb61/kube-multus/0.log"
Aug 05 17:42:52.279786 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:52.279757    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n6mhh_4a578ff9-e0d4-4386-a305-bcd010078bab/kube-multus-additional-cni-plugins/0.log"
Aug 05 17:42:52.307214 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:52.307190    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n6mhh_4a578ff9-e0d4-4386-a305-bcd010078bab/egress-router-binary-copy/0.log"
Aug 05 17:42:52.339113 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:52.339088    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n6mhh_4a578ff9-e0d4-4386-a305-bcd010078bab/cni-plugins/0.log"
Aug 05 17:42:52.372458 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:52.372428    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n6mhh_4a578ff9-e0d4-4386-a305-bcd010078bab/bond-cni-plugin/0.log"
Aug 05 17:42:52.398114 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:52.398063    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n6mhh_4a578ff9-e0d4-4386-a305-bcd010078bab/routeoverride-cni/0.log"
Aug 05 17:42:52.431187 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:52.431160    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n6mhh_4a578ff9-e0d4-4386-a305-bcd010078bab/whereabouts-cni-bincopy/0.log"
Aug 05 17:42:52.460748 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:52.460698    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-n6mhh_4a578ff9-e0d4-4386-a305-bcd010078bab/whereabouts-cni/0.log"
Aug 05 17:42:52.669410 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:52.669318    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xwnbn_a7531d8b-5f4c-43e0-bc2f-d9e2a415f660/network-metrics-daemon/0.log"
Aug 05 17:42:52.701180 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:52.701143    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xwnbn_a7531d8b-5f4c-43e0-bc2f-d9e2a415f660/kube-rbac-proxy/0.log"
Aug 05 17:42:53.974313 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:53.974280    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrhbh_03b7f246-02d1-4207-94ff-1264fb006a98/ovn-controller/0.log"
Aug 05 17:42:53.997403 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:53.997376    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrhbh_03b7f246-02d1-4207-94ff-1264fb006a98/ovn-acl-logging/0.log"
Aug 05 17:42:54.027796 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:54.027762    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrhbh_03b7f246-02d1-4207-94ff-1264fb006a98/kube-rbac-proxy-node/0.log"
Aug 05 17:42:54.051604 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:54.051577    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrhbh_03b7f246-02d1-4207-94ff-1264fb006a98/kube-rbac-proxy-ovn-metrics/0.log"
Aug 05 17:42:54.076498 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:54.076437    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrhbh_03b7f246-02d1-4207-94ff-1264fb006a98/northd/0.log"
Aug 05 17:42:54.105239 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:54.105209    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrhbh_03b7f246-02d1-4207-94ff-1264fb006a98/nbdb/0.log"
Aug 05 17:42:54.130142 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:54.130114    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrhbh_03b7f246-02d1-4207-94ff-1264fb006a98/sbdb/0.log"
Aug 05 17:42:54.196278 ip-10-0-142-192 kubenswrapper[2574]: I0805 17:42:54.196230    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xrhbh_03b7f246-02d1-4207-94ff-1264fb006a98/ovnkube-controller/0.log"