Sep 17 23:10:43.783110 ip-10-0-140-43 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 23:10:47.166127 ip-10-0-140-43 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:47.166127 ip-10-0-140-43 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 23:10:47.166127 ip-10-0-140-43 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:47.166127 ip-10-0-140-43 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 23:10:47.166127 ip-10-0-140-43 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:47.166823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.166533    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 23:10:47.169995 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.169979    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:47.169995 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.169995    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.169999    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170002    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170005    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170007    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170010    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170013    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170019    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170023    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170026    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170030    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170034    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170037    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170040    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170043    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170045    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170048    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170051    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:47.170057 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170054    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170056    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170059    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170062    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170065    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170068    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170070    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170073    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170076    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170078    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170081    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170083    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170086    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170088    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170091    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170093    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170096    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170098    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170100    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170103    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:47.170511 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170105    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170108    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170113    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170116    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170118    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170120    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170123    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170125    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170128    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170130    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170132    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170135    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170137    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170140    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170144    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170147    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170150    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170152    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170155    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170157    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:47.171037 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170159    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170163    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170167    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170170    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170173    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170175    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170178    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170180    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170183    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170185    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170188    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170191    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170194    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170196    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170199    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170204    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170208    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170211    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170214    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:47.171595 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170217    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170219    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170222    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170225    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170228    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170231    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170233    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170236    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170239    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170242    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170245    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170248    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170251    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170253    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170256    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170258    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170261    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170263    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170266    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170269    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:47.172339 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170271    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170972    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170979    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170982    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170985    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170988    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170992    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170996    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.170999    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171003    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171006    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171009    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171012    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171015    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171018    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171020    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171023    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171026    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171029    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:47.173086 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171032    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171035    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171038    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171040    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171043    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171046    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171048    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171051    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171054    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171059    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171061    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171064    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171066    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171069    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171071    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171074    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171076    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171079    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171082    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171084    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:47.173679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171087    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171089    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171091    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171094    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171097    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171099    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171102    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171104    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171107    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171109    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171112    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171115    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171117    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171120    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171123    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171126    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171128    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171132    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171136    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171139    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:47.174203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171141    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171146    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171149    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171152    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171154    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171157    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171160    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171162    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171165    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171168    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171170    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171173    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171175    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171177    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171180    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171186    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171189    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171191    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171194    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171196    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:47.174698 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171199    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171201    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171204    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171206    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171209    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171211    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171215    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171217    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171219    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171222    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171224    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171227    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171229    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171234    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171237    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171239    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171242    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171244    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171247    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:47.175218 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171250    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.171252    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173146    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173157    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173160    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173163    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173167    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173169    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173172    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173175    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173177    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173181    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173184    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173186    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173189    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173191    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173194    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173197    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173199    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173202    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:47.175679 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173205    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173207    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173211    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173216    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173219    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173222    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173224    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173227    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173229    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173231    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173235    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173239    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173242    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173245    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173248    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173251    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173253    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173256    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173258    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:47.176203 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173261    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173263    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173266    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173268    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173271    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173274    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173276    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173278    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173281    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173283    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173286    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173288    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173291    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173294    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173296    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173299    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173301    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173306    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173309    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173312    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:47.176663 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173315    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173318    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173321    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173324    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173326    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173329    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173332    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173336    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173338    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173341    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173344    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173346    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173349    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173351    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173354    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173356    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173360    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173362    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173364    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173367    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:47.177163 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173369    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173372    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173374    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173377    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173379    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173381    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173385    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173387    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173390    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173392    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173395    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173397    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173399    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173402    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173404    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173407    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173409    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173411    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173414    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:47.177673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173416    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173419    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.173422    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173490    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173525    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173532    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173537    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173541    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173544    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173548    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173552    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173555    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173558    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173562    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173565    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173568    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173571    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173574    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173577    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173580    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173583    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173587    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173590    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 23:10:47.178142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173593    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173596    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173599    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173604    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173606    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173609    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173612    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173615    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173618    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173621    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173624    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173627    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173632    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173635    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173637    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173640    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173643    2582 flags.go:64] FLAG: --enable-server="true"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173646    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173651    2582 flags.go:64] FLAG: --event-burst="100"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173654    2582 flags.go:64] FLAG: --event-qps="50"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173657    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173660    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173663    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173666    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 23:10:47.178710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173669    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173672    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173675    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173678    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173681    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173684    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173687    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173690    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173693    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173696    2582 flags.go:64] FLAG: --feature-gates=""
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173700    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173702    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173705    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173708    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173711    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173714    2582 flags.go:64] FLAG: --help="false"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173717    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173720    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173723    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173726    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173731    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173735    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173738    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 23:10:47.179353 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173741    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173744    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173747    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173749    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173752    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173755    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173758    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173761    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173763    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173766    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173769    2582 flags.go:64] FLAG: --lock-file=""
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173771    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173774    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173777    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173782    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173784    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173788    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173790    2582 flags.go:64] FLAG: --logging-format="text"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173793    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173796    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173799    2582 flags.go:64] FLAG: --manifest-url=""
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173801    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173805    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173808    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173812    2582 flags.go:64] FLAG: --max-pods="110"
Sep 17 23:10:47.179998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173814    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173817    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173823    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173826    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173830    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173849    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173853    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173860    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173863    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173866    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173869    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173872    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173878    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173881    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173884    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173887    2582 flags.go:64] FLAG: --port="10250"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173890    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173893    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-022a8052a6cc30aca"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173896    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173899    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173902    2582 flags.go:64] FLAG: --register-node="true"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173904    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173908    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173911    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 17 23:10:47.180687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173914    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173917    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173920    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173923    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173926    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173929    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173931    2582 flags.go:64] FLAG: --runonce="false"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173934    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173937    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173940    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173942    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173946    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173950    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173953    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173956    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173959    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173962    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173965    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173967    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173970    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173973    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173976    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173981    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173984    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173987    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 23:10:47.181308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173991    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173993    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173996    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.173999    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.174001    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.174004    2582 flags.go:64] FLAG: --v="2"
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.174008    2582 flags.go:64] FLAG: --version="false"
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.174012    2582 flags.go:64] FLAG: --vmodule=""
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.174016    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.174019    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174124    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174128    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174131    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174134    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174136    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174139    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174142    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174145    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174149    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174152    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174155    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174158    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:47.181930 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174160    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174163    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174165    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174170    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174173    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174176    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174179    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174181    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174184    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174186    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174189    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174192    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174194    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174197    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174200    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174202    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174205    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174208    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174210    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:47.182465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174214    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174216    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174219    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174221    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174224    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174226    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174229    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174231    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174234    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174238    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174242    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174245    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174249    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174252    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174255    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174258    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174262    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174265    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174268    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:47.182966 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174271    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174274    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174277    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174280    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174282    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174285    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174288    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174290    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174293    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174295    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174298    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174300    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174303    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174306    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174308    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174311    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174313    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174316    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174319    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174321    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:47.183436 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174324    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174326    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174330    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174334    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174337    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174339    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174342    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174345    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174347    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174350    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174352    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174355    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174357    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174360    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174362    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174365    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174367    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174370    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174372    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174375    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:47.184009 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174378    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:47.184518 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174380    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:47.184518 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174383    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:47.184518 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174385    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:47.184518 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174388    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:47.184518 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174390    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:47.184518 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174393    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:47.184518 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174395    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:47.184518 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.174398    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:47.184518 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.174946    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:47.184518 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.181718    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 23:10:47.184518 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.181732    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 23:10:47.184518 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.181751    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 23:10:47.184518 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.181756    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 23:10:47.184518 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181813    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181818    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181822    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181825    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181828    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181849    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181854    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181857    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181860    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181864    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181868    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181872    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181875    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181878    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181881    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181883    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181886    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181888    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181891    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181893    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:47.184920 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181896    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181898    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181901    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181903    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181906    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181910    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181914    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181917    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181920    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181924    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181928    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181931    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181934    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181937    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181940    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181943    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181946    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181948    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181952    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:47.185394 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181955    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181958    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181962    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181965    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181967    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181970    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181972    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181980    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181983    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181985    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181988    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181990    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181993    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181995    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.181998    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182001    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182003    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182006    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182009    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182012    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:47.185854 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182015    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182017    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182020    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182024    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182026    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182029    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182032    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182034    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182037    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182039    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182042    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182044    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182047    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182049    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182052    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182054    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182056    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182059    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182062    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182064    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:47.186383 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182067    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182070    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182072    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182075    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182078    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182080    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182083    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182085    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182088    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182090    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182093    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182095    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182098    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182101    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182105    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182107    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182110    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182112    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182115    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:47.186898 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182117    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:47.187351 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.182122    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:47.187351 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182256    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:47.187351 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182261    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:47.187351 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182264    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:47.187351 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182267    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:47.187351 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182270    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:47.187351 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182273    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:47.187351 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182276    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:47.187351 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182279    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:47.187351 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182281    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:47.187351 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182284    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:47.187351 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182286    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:47.187351 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182289    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:47.187351 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182292    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:47.187351 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182294    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182296    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182299    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182301    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182304    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182306    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182310    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182314    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182317    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182320    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182322    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182326    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182329    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182332    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182334    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182337    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182339    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182342    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182344    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:47.187731 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182347    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182349    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182352    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182354    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182357    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182359    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182361    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182364    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182366    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182369    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182371    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182373    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182376    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182378    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182381    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182383    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182385    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182388    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182390    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:47.188230 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182392    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182395    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182397    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182400    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182402    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182405    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182408    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182411    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182413    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182416    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182418    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182420    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182423    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182425    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182428    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182430    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182433    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182436    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182439    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182443    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:47.188688 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182446    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182448    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182451    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182453    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182456    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182458    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182461    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182464    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182466    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182469    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182472    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182475    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182478    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182480    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182483    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182485    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182488    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182490    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182493    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182497    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:47.189737 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182501    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:47.193643 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182503    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:47.193643 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182506    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:47.193643 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182508    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:47.193643 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182511    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:47.193643 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182513    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:47.193643 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182516    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:47.193643 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:47.182518    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:47.193643 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.182522    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:47.193643 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.182629    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 23:10:47.193643 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.184388    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 23:10:47.193643 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.185200    2582 server.go:1019] "Starting client certificate rotation"
Sep 17 23:10:47.193643 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.185300    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:10:47.193643 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.185331    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:10:47.193991 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.187664    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:10:47.193991 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.189514    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:10:47.195468 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.195448    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 17 23:10:47.197087 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.197074    2582 log.go:25] "Validated CRI v1 image API"
Sep 17 23:10:47.197922 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.197903    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 23:10:47.202131 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.202108    2582 fs.go:135] Filesystem UUIDs: map[22b1d93a-0db0-4d68-9fc7-19afb2ad6376:/dev/nvme0n1p4 300a1c48-fa6a-4b17-bc93-589667f5cc37:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 17 23:10:47.202195 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.202131    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 23:10:47.207776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.207672    2582 manager.go:216] Machine: {Timestamp:2026-09-17 23:10:47.20720487 +0000 UTC m=+1.894277438 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100412 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a21c53130fd9e7bbd2650081a99be SystemUUID:ec2a21c5-3130-fd9e-7bbd-2650081a99be BootID:29949c34-e577-488d-8c4f-710d1dda7669 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:54:22:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:54:22:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:25:21:62:9f:d1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 23:10:47.207776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.207774    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 23:10:47.207891 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.207863    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 23:10:47.209264 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.209241    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 23:10:47.209384 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.209267    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-43.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 23:10:47.210270 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.210261    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 23:10:47.210308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.210272    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 23:10:47.210308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.210280    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:10:47.210308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.210290    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:10:47.210385 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.210356    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 23:10:47.213500 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.213489    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:10:47.213612 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.213603    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 23:10:47.213648 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.213636    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 23:10:47.213648 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.213646    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 23:10:47.213701 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.213659    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 23:10:47.213701 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.213668    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 17 23:10:47.213701 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.213677    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 23:10:47.213999 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.213987    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:10:47.214040 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.214006    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:10:47.217055 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.217034    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 23:10:47.219024 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.219010    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 23:10:47.219068 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.219041    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 23:10:47.219972 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.219961    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 23:10:47.220005 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.219978    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 23:10:47.220005 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.219984    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 23:10:47.220005 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.219989    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 23:10:47.220005 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.219998    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 23:10:47.220005 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.220003    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 23:10:47.220133 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.220017    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 23:10:47.220133 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.220022    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 23:10:47.220133 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.220029    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 23:10:47.220133 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.220034    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 23:10:47.220133 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.220043    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 23:10:47.220133 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.220047    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 23:10:47.220133 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.220073    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 23:10:47.220133 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.220078    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 23:10:47.223444 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.223431    2582 server.go:1268] "Started kubelet"
Sep 17 23:10:47.223798 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.223779    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 23:10:47.223862 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.223813    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 23:10:47.224088 ip-10-0-140-43 systemd[1]: Started Kubernetes Kubelet.
Sep 17 23:10:47.224562 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.224129    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 23:10:47.225617 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.225602    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 23:10:47.226064 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.226043    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:10:47.226502 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.226480    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 23:10:47.229408 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.229389    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 23:10:47.254294 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.254271    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 23:10:47.254466 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.254446    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 23:10:47.255120 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.255100    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found"
Sep 17 23:10:47.255232 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.255215    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 23:10:47.255285 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.255238    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 23:10:47.255285 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.255264    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 23:10:47.255387 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.255375    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 23:10:47.255461 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.255388    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 23:10:47.255857 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.255822    2582 factory.go:55] Registering systemd factory
Sep 17 23:10:47.255956 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.255944    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 17 23:10:47.256319 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.256299    2582 factory.go:153] Registering CRI-O factory
Sep 17 23:10:47.256319 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.256322    2582 factory.go:223] Registration of the crio container factory successfully
Sep 17 23:10:47.256446 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.256426    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 23:10:47.256499 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.256451    2582 factory.go:103] Registering Raw factory
Sep 17 23:10:47.256499 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.256470    2582 manager.go:1196] Started watching for new ooms in manager
Sep 17 23:10:47.257486 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.257471    2582 manager.go:349] Starting recovery of all containers
Sep 17 23:10:47.259901 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.259882    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-43.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 23:10:47.260136 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.260109    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 23:10:47.260225 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.260206    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 23:10:47.260431 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.260407    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 23:10:47.260519 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.260479    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 23:10:47.260968 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.259929    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-43.ec2.internal.18d63e860302c7f5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-43.ec2.internal,UID:ip-10-0-140-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-43.ec2.internal,},FirstTimestamp:2026-09-17 23:10:47.223404533 +0000 UTC m=+1.910477099,LastTimestamp:2026-09-17 23:10:47.223404533 +0000 UTC m=+1.910477099,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-43.ec2.internal,}"
Sep 17 23:10:47.267148 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.267132    2582 manager.go:354] Recovery completed
Sep 17 23:10:47.270952 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.270941    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:47.272797 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.272776    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:47.272888 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.272809    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:47.272888 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.272819    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:47.273342 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.273326    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 23:10:47.273342 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.273338    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 23:10:47.273446 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.273356    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:10:47.273446 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.273339    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:47.274034 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.274016    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:47.274099 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.274049    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:47.274099 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.274064    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:47.275669 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.275654    2582 policy_none.go:49] "None policy: Start"
Sep 17 23:10:47.275669 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.275671    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 23:10:47.275774 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.275681    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 23:10:47.277928 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.277916    2582 policy_none.go:47] "Start"
Sep 17 23:10:47.284261 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.284194    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-43.ec2.internal.18d63e8605f4732d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-43.ec2.internal,UID:ip-10-0-140-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-43.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-43.ec2.internal,},FirstTimestamp:2026-09-17 23:10:47.272796973 +0000 UTC m=+1.959869541,LastTimestamp:2026-09-17 23:10:47.272796973 +0000 UTC m=+1.959869541,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-43.ec2.internal,}"
Sep 17 23:10:47.296896 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.296574    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-43.ec2.internal.18d63e8605f4b389  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-43.ec2.internal,UID:ip-10-0-140-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-43.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-43.ec2.internal,},FirstTimestamp:2026-09-17 23:10:47.272813449 +0000 UTC m=+1.959886017,LastTimestamp:2026-09-17 23:10:47.272813449 +0000 UTC m=+1.959886017,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-43.ec2.internal,}"
Sep 17 23:10:47.309048 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.308984    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-43.ec2.internal.18d63e8605f4d661  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-43.ec2.internal,UID:ip-10-0-140-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-43.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-43.ec2.internal,},FirstTimestamp:2026-09-17 23:10:47.272822369 +0000 UTC m=+1.959894937,LastTimestamp:2026-09-17 23:10:47.272822369 +0000 UTC m=+1.959894937,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-43.ec2.internal,}"
Sep 17 23:10:47.332028 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.321037    2582 manager.go:337] "Starting Device Plugin manager"
Sep 17 23:10:47.332028 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.321072    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 23:10:47.332028 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.321081    2582 server.go:87] "Starting device plugin registration server"
Sep 17 23:10:47.332028 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.321297    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 23:10:47.332028 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.321320    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 23:10:47.332028 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.321394    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 23:10:47.332028 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.321480    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 23:10:47.332028 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.321489    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 23:10:47.332028 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.321960    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 23:10:47.332028 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.322025    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-43.ec2.internal\" not found"
Sep 17 23:10:47.332028 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.329734    2582 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-43.ec2.internal.18d63e8605f4732d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-43.ec2.internal.18d63e8605f4732d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-43.ec2.internal,UID:ip-10-0-140-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-43.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-43.ec2.internal,},FirstTimestamp:2026-09-17 23:10:47.272796973 +0000 UTC m=+1.959869541,LastTimestamp:2026-09-17 23:10:47.274032245 +0000 UTC m=+1.961105006,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-43.ec2.internal,}"
Sep 17 23:10:47.342556 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.342401    2582 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-43.ec2.internal.18d63e8605f4b389\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-43.ec2.internal.18d63e8605f4b389  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-43.ec2.internal,UID:ip-10-0-140-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-43.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-43.ec2.internal,},FirstTimestamp:2026-09-17 23:10:47.272813449 +0000 UTC m=+1.959886017,LastTimestamp:2026-09-17 23:10:47.274056502 +0000 UTC m=+1.961129074,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-43.ec2.internal,}"
Sep 17 23:10:47.354428 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.354373    2582 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-43.ec2.internal.18d63e8605f4d661\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-43.ec2.internal.18d63e8605f4d661  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-43.ec2.internal,UID:ip-10-0-140-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-43.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-43.ec2.internal,},FirstTimestamp:2026-09-17 23:10:47.272822369 +0000 UTC m=+1.959894937,LastTimestamp:2026-09-17 23:10:47.274070603 +0000 UTC m=+1.961143175,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-43.ec2.internal,}"
Sep 17 23:10:47.375710 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.375651    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-43.ec2.internal.18d63e86098ed5d2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-43.ec2.internal,UID:ip-10-0-140-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-140-43.ec2.internal,},FirstTimestamp:2026-09-17 23:10:47.333246418 +0000 UTC m=+2.020318980,LastTimestamp:2026-09-17 23:10:47.333246418 +0000 UTC m=+2.020318980,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-43.ec2.internal,}"
Sep 17 23:10:47.386676 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.386652    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 23:10:47.387982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.387961    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 23:10:47.388066 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.388006    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 23:10:47.388066 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.388025    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 23:10:47.388152 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.388060    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 23:10:47.400305 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.400280    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 17 23:10:47.416354 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.416314    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-knvgl"
Sep 17 23:10:47.422371 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.422354    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:47.423098 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.423077    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:47.423179 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.423105    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:47.423179 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.423116    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:47.423179 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.423152    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.429499 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.429485    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-knvgl"
Sep 17 23:10:47.482759 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.482740    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.482831 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.482766    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-43.ec2.internal\": node \"ip-10-0-140-43.ec2.internal\" not found"
Sep 17 23:10:47.488749 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.488726    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal"]
Sep 17 23:10:47.488828 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.488783    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:47.490090 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.490074    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:47.490159 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.490103    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:47.490159 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.490114    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:47.492134 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.492123    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:47.492249 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.492230    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.492321 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.492258    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:47.492769 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.492752    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:47.492858 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.492778    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:47.492858 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.492793    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:47.492941 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.492860    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:47.492941 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.492879    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:47.492941 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.492889    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:47.494878 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.494866    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.494919 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.494890    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:47.495532 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.495515    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:47.495589 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.495549    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:47.495589 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.495564    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:47.521438 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.521417    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-43.ec2.internal\" not found" node="ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.525680 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.525666    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-43.ec2.internal\" not found" node="ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.606480 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.606456    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found"
Sep 17 23:10:47.656426 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.656403    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8486b3e20df1174f1076e7f4de44e909-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal\" (UID: \"8486b3e20df1174f1076e7f4de44e909\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.656518 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.656433    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8486b3e20df1174f1076e7f4de44e909-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal\" (UID: \"8486b3e20df1174f1076e7f4de44e909\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.656518 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.656457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad31e4fc899beb2f71ca79970426a20a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-43.ec2.internal\" (UID: \"ad31e4fc899beb2f71ca79970426a20a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.707054 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.707012    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found"
Sep 17 23:10:47.756923 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.756903    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8486b3e20df1174f1076e7f4de44e909-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal\" (UID: \"8486b3e20df1174f1076e7f4de44e909\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.756978 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.756955    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8486b3e20df1174f1076e7f4de44e909-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal\" (UID: \"8486b3e20df1174f1076e7f4de44e909\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.757018 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.756994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8486b3e20df1174f1076e7f4de44e909-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal\" (UID: \"8486b3e20df1174f1076e7f4de44e909\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.757052 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.757017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad31e4fc899beb2f71ca79970426a20a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-43.ec2.internal\" (UID: \"ad31e4fc899beb2f71ca79970426a20a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.757090 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.757050    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad31e4fc899beb2f71ca79970426a20a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-43.ec2.internal\" (UID: \"ad31e4fc899beb2f71ca79970426a20a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.757090 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.757077    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8486b3e20df1174f1076e7f4de44e909-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal\" (UID: \"8486b3e20df1174f1076e7f4de44e909\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.807947 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.807928    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found"
Sep 17 23:10:47.824406 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.824388    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.828424 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:47.828407    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal"
Sep 17 23:10:47.908691 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:47.908643    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found"
Sep 17 23:10:48.009135 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:48.009071    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found"
Sep 17 23:10:48.109595 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:48.109570    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found"
Sep 17 23:10:48.185170 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.185152    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 23:10:48.210449 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:48.210427    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found"
Sep 17 23:10:48.311259 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:48.311240    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found"
Sep 17 23:10:48.412155 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:48.412128    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found"
Sep 17 23:10:48.423888 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.423866    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:48.434747 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.434712    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 23:05:47 +0000 UTC" deadline="2028-02-21 16:28:41.458764453 +0000 UTC"
Sep 17 23:10:48.434747 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.434760    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12521h17m53.024024976s"
Sep 17 23:10:48.483256 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.483237    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:48.484766 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.484748    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:48.484885 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.484780    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:48.484885 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.484789    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:48.486497 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:48.486471    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8486b3e20df1174f1076e7f4de44e909.slice/crio-72f69bd49d68a40d4349e782f1cbc5552485cf3ad36ce060f500e4b2ca8d393a WatchSource:0}: Error finding container 72f69bd49d68a40d4349e782f1cbc5552485cf3ad36ce060f500e4b2ca8d393a: Status 404 returned error can't find the container with id 72f69bd49d68a40d4349e782f1cbc5552485cf3ad36ce060f500e4b2ca8d393a
Sep 17 23:10:48.487055 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:48.487036    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad31e4fc899beb2f71ca79970426a20a.slice/crio-5a29c9c01650935f21653ef3f3de7b7c285593568f558f0ff610dc2086a279bd WatchSource:0}: Error finding container 5a29c9c01650935f21653ef3f3de7b7c285593568f558f0ff610dc2086a279bd: Status 404 returned error can't find the container with id 5a29c9c01650935f21653ef3f3de7b7c285593568f558f0ff610dc2086a279bd
Sep 17 23:10:48.491796 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.491781    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:10:48.513067 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:48.513049    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-43.ec2.internal\" not found"
Sep 17 23:10:48.551796 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.551752    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:48.592454 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.592436    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:48.636031 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.636012    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:48.655393 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.655377    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal"
Sep 17 23:10:48.666436 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.666420    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:10:48.667108 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.667097    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal"
Sep 17 23:10:48.679927 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:48.679909    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:10:49.215598 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.215575    2582 apiserver.go:52] "Watching apiserver"
Sep 17 23:10:49.229643 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.229620    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 23:10:49.230514 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.230481    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-s7lg2","openshift-cluster-node-tuning-operator/tuned-nbvc8","openshift-image-registry/node-ca-pxhs2","openshift-multus/multus-t2x98","openshift-network-diagnostics/network-check-target-5xnb7","openshift-ovn-kubernetes/ovnkube-node-88d87","kube-system/konnectivity-agent-mzjpw","kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w","openshift-dns/node-resolver-95nk9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal","openshift-multus/multus-additional-cni-plugins-j6vfx","openshift-multus/network-metrics-daemon-nmn9p","openshift-network-operator/iptables-alerter-d9hfz"]
Sep 17 23:10:49.233130 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.233109    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mzjpw"
Sep 17 23:10:49.235383 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.235360    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:10:49.235479 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.235424    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:10:49.236104 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.236084    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 23:10:49.236589 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.236394    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 23:10:49.236829 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.236813    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vjvs5\""
Sep 17 23:10:49.237494 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.237478    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pxhs2"
Sep 17 23:10:49.241788 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.239809    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.242175 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.242153    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 23:10:49.242273 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.242174    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 23:10:49.242332 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.242308    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 23:10:49.242450 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.242423    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hzpmp\""
Sep 17 23:10:49.242535 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.242459    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 23:10:49.243206 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.243187    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 23:10:49.243405 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.243388    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 23:10:49.243480 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.243448    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2ljbs\""
Sep 17 23:10:49.243589 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.243568    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:10:49.243665 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.243642    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:10:49.245542 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.245521    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.246060 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.246041    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 23:10:49.248010 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.247990    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 23:10:49.248100 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.248034    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.248726 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.248707    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gfdn5\""
Sep 17 23:10:49.248909 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.248894    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 23:10:49.249323 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.249177    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 23:10:49.249387 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.249341    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 23:10:49.249899 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.249873    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 23:10:49.249977 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.249871    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 23:10:49.250425 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.250407    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:49.250499 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.250478    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:10:49.250739 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.250723    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 23:10:49.250962 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.250944    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 23:10:49.251293 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.251274    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 23:10:49.251619 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.251603    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w9xh8\""
Sep 17 23:10:49.253014 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.252980    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.255812 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.255792    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-95nk9"
Sep 17 23:10:49.256189 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.256174    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 23:10:49.256567 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.256549    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:49.256950 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.256931    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4tdcq\""
Sep 17 23:10:49.257123 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.257065    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:49.258366 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.257937    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d9hfz"
Sep 17 23:10:49.258671 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.258652    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 23:10:49.259004 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.258988    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 23:10:49.259388 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.259374    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b8h9b\""
Sep 17 23:10:49.260767 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.260750    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.261481 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.261464    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:49.261783 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.261767    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 23:10:49.261909 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.261773    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:49.261966 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.261929    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t2p6k\""
Sep 17 23:10:49.263821 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.263664    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s8vjw\""
Sep 17 23:10:49.264161 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264146    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 23:10:49.264299 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-env-overrides\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.264299 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264298    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-modprobe-d\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.264445 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264314    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-sys\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.264445 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264327    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b67fba1-4bba-4dc6-b35f-a9e3375ffd79-host-slash\") pod \"iptables-alerter-d9hfz\" (UID: \"5b67fba1-4bba-4dc6-b35f-a9e3375ffd79\") " pod="openshift-network-operator/iptables-alerter-d9hfz"
Sep 17 23:10:49.264445 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264346    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-multus-cni-dir\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.264445 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264180    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 23:10:49.264445 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-run-k8s-cni-cncf-io\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.264445 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-etc-openvswitch\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.264445 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-sysctl-d\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.264776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xftvw\" (UniqueName: \"kubernetes.io/projected/de993612-482a-4476-a44b-cef040d77485-kube-api-access-xftvw\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.264776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-systemd-units\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.264776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-ovnkube-script-lib\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.264776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264549    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-var-lib-kubelet\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.264776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b41f4c0-9428-4c24-b1bc-725169912218-konnectivity-ca\") pod \"konnectivity-agent-mzjpw\" (UID: \"0b41f4c0-9428-4c24-b1bc-725169912218\") " pod="kube-system/konnectivity-agent-mzjpw"
Sep 17 23:10:49.264776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264597    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-multus-socket-dir-parent\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.264776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-multus-conf-dir\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.264776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264642    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-etc-kubernetes\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.264776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264671    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.264776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-sysconfig\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.264776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-systemd\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.264776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264747    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0fe30d8-d67c-4736-86fe-e33d93dafbb0-hosts-file\") pod \"node-resolver-95nk9\" (UID: \"d0fe30d8-d67c-4736-86fe-e33d93dafbb0\") " pod="openshift-dns/node-resolver-95nk9"
Sep 17 23:10:49.264776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b67fba1-4bba-4dc6-b35f-a9e3375ffd79-iptables-alerter-script\") pod \"iptables-alerter-d9hfz\" (UID: \"5b67fba1-4bba-4dc6-b35f-a9e3375ffd79\") " pod="openshift-network-operator/iptables-alerter-d9hfz"
Sep 17 23:10:49.265218 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264793    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-var-lib-cni-multus\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.265218 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-run-multus-certs\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.265218 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-registration-dir\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.265218 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264901    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de993612-482a-4476-a44b-cef040d77485-tmp\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.265218 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-lib-modules\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.265218 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264945    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de993612-482a-4476-a44b-cef040d77485-etc-tuned\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.265218 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.264979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njj27\" (UniqueName: \"kubernetes.io/projected/d0fe30d8-d67c-4736-86fe-e33d93dafbb0-kube-api-access-njj27\") pod \"node-resolver-95nk9\" (UID: \"d0fe30d8-d67c-4736-86fe-e33d93dafbb0\") " pod="openshift-dns/node-resolver-95nk9"
Sep 17 23:10:49.265218 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5z7r\" (UniqueName: \"kubernetes.io/projected/5b67fba1-4bba-4dc6-b35f-a9e3375ffd79-kube-api-access-x5z7r\") pod \"iptables-alerter-d9hfz\" (UID: \"5b67fba1-4bba-4dc6-b35f-a9e3375ffd79\") " pod="openshift-network-operator/iptables-alerter-d9hfz"
Sep 17 23:10:49.265218 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265036    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-cnibin\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.265218 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265059    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-log-socket\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.265218 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265159    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-ovn-node-metrics-cert\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.265218 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-socket-dir\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.265218 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265209    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265231    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-run-netns\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265254    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-run-netns\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265275    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-run-ovn\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265305    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-device-dir\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-run\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265360    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b41f4c0-9428-4c24-b1bc-725169912218-agent-certs\") pod \"konnectivity-agent-mzjpw\" (UID: \"0b41f4c0-9428-4c24-b1bc-725169912218\") " pod="kube-system/konnectivity-agent-mzjpw"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265383    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-var-lib-cni-bin\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265421    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/473f1449-5a91-400d-8225-88e8716cd98d-multus-daemon-config\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-run-ovn-kubernetes\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fe30d8-d67c-4736-86fe-e33d93dafbb0-tmp-dir\") pod \"node-resolver-95nk9\" (UID: \"d0fe30d8-d67c-4736-86fe-e33d93dafbb0\") " pod="openshift-dns/node-resolver-95nk9"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265486    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70ea03b1-9946-46d6-b68a-9c730d932c72-serviceca\") pod \"node-ca-pxhs2\" (UID: \"70ea03b1-9946-46d6-b68a-9c730d932c72\") " pod="openshift-image-registry/node-ca-pxhs2"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265503    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-system-cni-dir\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-var-lib-kubelet\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265536    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-run-systemd\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265555    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-var-lib-openvswitch\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs\") pod \"network-metrics-daemon-nmn9p\" (UID: \"80efd7c7-aa08-47ee-a917-251ef8624493\") " pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:10:49.265823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d48fd517-45d0-445c-836e-7045570e3385-dbus\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265603    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-kubelet\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf6xh\" (UniqueName: \"kubernetes.io/projected/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-kube-api-access-mf6xh\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265699    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-sysctl-conf\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265720    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-host\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d48fd517-45d0-445c-836e-7045570e3385-kubelet-config\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-hostroot\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265850    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-slash\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-run-openvswitch\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265918    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-kubernetes\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265965    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70ea03b1-9946-46d6-b68a-9c730d932c72-host\") pod \"node-ca-pxhs2\" (UID: \"70ea03b1-9946-46d6-b68a-9c730d932c72\") " pod="openshift-image-registry/node-ca-pxhs2"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.265996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhczf\" (UniqueName: \"kubernetes.io/projected/70ea03b1-9946-46d6-b68a-9c730d932c72-kube-api-access-zhczf\") pod \"node-ca-pxhs2\" (UID: \"70ea03b1-9946-46d6-b68a-9c730d932c72\") " pod="openshift-image-registry/node-ca-pxhs2"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.266026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-cni-bin\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.266084    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd8sh\" (UniqueName: \"kubernetes.io/projected/80efd7c7-aa08-47ee-a917-251ef8624493-kube-api-access-hd8sh\") pod \"network-metrics-daemon-nmn9p\" (UID: \"80efd7c7-aa08-47ee-a917-251ef8624493\") " pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.266117    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-node-log\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.266528 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.266130    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-ovnkube-config\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.267099 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.266143    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctkqc\" (UniqueName: \"kubernetes.io/projected/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-kube-api-access-ctkqc\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.267099 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.266161    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-etc-selinux\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.267099 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.266196    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-sys-fs\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.267099 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.266238    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/473f1449-5a91-400d-8225-88e8716cd98d-cni-binary-copy\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.267099 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.266262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb68h\" (UniqueName: \"kubernetes.io/projected/473f1449-5a91-400d-8225-88e8716cd98d-kube-api-access-fb68h\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.267099 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.266287    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9tjl\" (UniqueName: \"kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl\") pod \"network-check-target-5xnb7\" (UID: \"9e2add4f-c624-4d66-a7c5-8136a8ff8943\") " pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:10:49.267099 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.266312    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-cni-netd\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.269851 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.269817    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:10:49.315257 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.315236    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lchcv"
Sep 17 23:10:49.327639 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.327620    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lchcv"
Sep 17 23:10:49.356319 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.356296    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 23:10:49.367073 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367043    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d48fd517-45d0-445c-836e-7045570e3385-dbus\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:49.367196 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367084    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/720f72ab-3b57-4871-9d92-4f546cf4217f-cni-binary-copy\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.367196 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367116    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/720f72ab-3b57-4871-9d92-4f546cf4217f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.367196 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-kubelet\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.367403 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf6xh\" (UniqueName: \"kubernetes.io/projected/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-kube-api-access-mf6xh\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.367403 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-sysctl-conf\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.367403 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367244    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-kubelet\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.367403 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367285    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d48fd517-45d0-445c-836e-7045570e3385-dbus\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:49.367403 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367297    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-host\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.367403 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-host\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.367403 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d48fd517-45d0-445c-836e-7045570e3385-kubelet-config\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:49.367403 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367350    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-hostroot\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.367403 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367374    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/720f72ab-3b57-4871-9d92-4f546cf4217f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.367403 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367389    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-sysctl-conf\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.367403 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-slash\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-run-openvswitch\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367434    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-hostroot\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367441    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d48fd517-45d0-445c-836e-7045570e3385-kubelet-config\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367482    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-slash\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-kubernetes\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367557    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-run-openvswitch\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70ea03b1-9946-46d6-b68a-9c730d932c72-host\") pod \"node-ca-pxhs2\" (UID: \"70ea03b1-9946-46d6-b68a-9c730d932c72\") " pod="openshift-image-registry/node-ca-pxhs2"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367595    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhczf\" (UniqueName: \"kubernetes.io/projected/70ea03b1-9946-46d6-b68a-9c730d932c72-kube-api-access-zhczf\") pod \"node-ca-pxhs2\" (UID: \"70ea03b1-9946-46d6-b68a-9c730d932c72\") " pod="openshift-image-registry/node-ca-pxhs2"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/720f72ab-3b57-4871-9d92-4f546cf4217f-system-cni-dir\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367658    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-cni-bin\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367654    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-kubernetes\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367692    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70ea03b1-9946-46d6-b68a-9c730d932c72-host\") pod \"node-ca-pxhs2\" (UID: \"70ea03b1-9946-46d6-b68a-9c730d932c72\") " pod="openshift-image-registry/node-ca-pxhs2"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367777    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-cni-bin\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd8sh\" (UniqueName: \"kubernetes.io/projected/80efd7c7-aa08-47ee-a917-251ef8624493-kube-api-access-hd8sh\") pod \"network-metrics-daemon-nmn9p\" (UID: \"80efd7c7-aa08-47ee-a917-251ef8624493\") " pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:10:49.367868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2v98\" (UniqueName: \"kubernetes.io/projected/720f72ab-3b57-4871-9d92-4f546cf4217f-kube-api-access-w2v98\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.368640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367870    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-node-log\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.368640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367893    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-ovnkube-config\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.368640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367954    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctkqc\" (UniqueName: \"kubernetes.io/projected/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-kube-api-access-ctkqc\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.368640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.367990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-etc-selinux\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.368640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-sys-fs\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.368640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/473f1449-5a91-400d-8225-88e8716cd98d-cni-binary-copy\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.368640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368056    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-node-log\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.368640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb68h\" (UniqueName: \"kubernetes.io/projected/473f1449-5a91-400d-8225-88e8716cd98d-kube-api-access-fb68h\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.368640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368346    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-ovnkube-config\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.368640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368355    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-etc-selinux\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.368640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368406    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-sys-fs\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.368640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tjl\" (UniqueName: \"kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl\") pod \"network-check-target-5xnb7\" (UID: \"9e2add4f-c624-4d66-a7c5-8136a8ff8943\") " pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:10:49.368640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368487    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-cni-netd\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.368640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-env-overrides\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-modprobe-d\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-sys\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368720    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-cni-netd\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368735    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b67fba1-4bba-4dc6-b35f-a9e3375ffd79-host-slash\") pod \"iptables-alerter-d9hfz\" (UID: \"5b67fba1-4bba-4dc6-b35f-a9e3375ffd79\") " pod="openshift-network-operator/iptables-alerter-d9hfz"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-multus-cni-dir\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368783    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-run-k8s-cni-cncf-io\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-etc-openvswitch\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368849    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-sysctl-d\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368876    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/473f1449-5a91-400d-8225-88e8716cd98d-cni-binary-copy\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xftvw\" (UniqueName: \"kubernetes.io/projected/de993612-482a-4476-a44b-cef040d77485-kube-api-access-xftvw\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-systemd-units\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.368980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-ovnkube-script-lib\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369004    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-var-lib-kubelet\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369019    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-sysctl-d\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369030    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-env-overrides\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b41f4c0-9428-4c24-b1bc-725169912218-konnectivity-ca\") pod \"konnectivity-agent-mzjpw\" (UID: \"0b41f4c0-9428-4c24-b1bc-725169912218\") " pod="kube-system/konnectivity-agent-mzjpw"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369074    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-multus-cni-dir\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.369364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369111    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-run-k8s-cni-cncf-io\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369128    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-modprobe-d\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369144    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-etc-openvswitch\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369158    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-sys\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369180    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b67fba1-4bba-4dc6-b35f-a9e3375ffd79-host-slash\") pod \"iptables-alerter-d9hfz\" (UID: \"5b67fba1-4bba-4dc6-b35f-a9e3375ffd79\") " pod="openshift-network-operator/iptables-alerter-d9hfz"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369204    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-multus-socket-dir-parent\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-multus-conf-dir\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369306    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-etc-kubernetes\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/720f72ab-3b57-4871-9d92-4f546cf4217f-cnibin\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369439    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-sysconfig\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-systemd\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0fe30d8-d67c-4736-86fe-e33d93dafbb0-hosts-file\") pod \"node-resolver-95nk9\" (UID: \"d0fe30d8-d67c-4736-86fe-e33d93dafbb0\") " pod="openshift-dns/node-resolver-95nk9"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b67fba1-4bba-4dc6-b35f-a9e3375ffd79-iptables-alerter-script\") pod \"iptables-alerter-d9hfz\" (UID: \"5b67fba1-4bba-4dc6-b35f-a9e3375ffd79\") " pod="openshift-network-operator/iptables-alerter-d9hfz"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369537    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-var-lib-cni-multus\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-run-multus-certs\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369572    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b41f4c0-9428-4c24-b1bc-725169912218-konnectivity-ca\") pod \"konnectivity-agent-mzjpw\" (UID: \"0b41f4c0-9428-4c24-b1bc-725169912218\") " pod="kube-system/konnectivity-agent-mzjpw"
Sep 17 23:10:49.370220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369587    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-registration-dir\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369626    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-multus-socket-dir-parent\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369654    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-systemd-units\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369667    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-run-multus-certs\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369667    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-ovnkube-script-lib\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369680    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de993612-482a-4476-a44b-cef040d77485-tmp\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-lib-modules\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de993612-482a-4476-a44b-cef040d77485-etc-tuned\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369724    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-registration-dir\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369731    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-var-lib-kubelet\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njj27\" (UniqueName: \"kubernetes.io/projected/d0fe30d8-d67c-4736-86fe-e33d93dafbb0-kube-api-access-njj27\") pod \"node-resolver-95nk9\" (UID: \"d0fe30d8-d67c-4736-86fe-e33d93dafbb0\") " pod="openshift-dns/node-resolver-95nk9"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369773    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-multus-conf-dir\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5z7r\" (UniqueName: \"kubernetes.io/projected/5b67fba1-4bba-4dc6-b35f-a9e3375ffd79-kube-api-access-x5z7r\") pod \"iptables-alerter-d9hfz\" (UID: \"5b67fba1-4bba-4dc6-b35f-a9e3375ffd79\") " pod="openshift-network-operator/iptables-alerter-d9hfz"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369824    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-etc-kubernetes\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.369959    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-lib-modules\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370099    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370312    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b67fba1-4bba-4dc6-b35f-a9e3375ffd79-iptables-alerter-script\") pod \"iptables-alerter-d9hfz\" (UID: \"5b67fba1-4bba-4dc6-b35f-a9e3375ffd79\") " pod="openshift-network-operator/iptables-alerter-d9hfz"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370368    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-sysconfig\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.371000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370396    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0fe30d8-d67c-4736-86fe-e33d93dafbb0-hosts-file\") pod \"node-resolver-95nk9\" (UID: \"d0fe30d8-d67c-4736-86fe-e33d93dafbb0\") " pod="openshift-dns/node-resolver-95nk9"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370401    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-var-lib-cni-multus\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370468    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-etc-systemd\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370465    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370505    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-cnibin\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370586    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-log-socket\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370602    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-cnibin\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370650    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-ovn-node-metrics-cert\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370677    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-socket-dir\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370709    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-log-socket\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-run-netns\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370812    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-run-netns\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-run-ovn\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-device-dir\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370923    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-run\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b41f4c0-9428-4c24-b1bc-725169912218-agent-certs\") pod \"konnectivity-agent-mzjpw\" (UID: \"0b41f4c0-9428-4c24-b1bc-725169912218\") " pod="kube-system/konnectivity-agent-mzjpw"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.370994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-var-lib-cni-bin\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.371982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.371012    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/473f1449-5a91-400d-8225-88e8716cd98d-multus-daemon-config\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.371037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/720f72ab-3b57-4871-9d92-4f546cf4217f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.371063    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-run-ovn-kubernetes\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.371079    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fe30d8-d67c-4736-86fe-e33d93dafbb0-tmp-dir\") pod \"node-resolver-95nk9\" (UID: \"d0fe30d8-d67c-4736-86fe-e33d93dafbb0\") " pod="openshift-dns/node-resolver-95nk9"
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.371094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70ea03b1-9946-46d6-b68a-9c730d932c72-serviceca\") pod \"node-ca-pxhs2\" (UID: \"70ea03b1-9946-46d6-b68a-9c730d932c72\") " pod="openshift-image-registry/node-ca-pxhs2"
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.371109    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-system-cni-dir\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.371162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-var-lib-kubelet\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.371186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-run-systemd\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.371212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-var-lib-openvswitch\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.371236    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs\") pod \"network-metrics-daemon-nmn9p\" (UID: \"80efd7c7-aa08-47ee-a917-251ef8624493\") " pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.371405    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.371483    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs podName:80efd7c7-aa08-47ee-a917-251ef8624493 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:49.871452748 +0000 UTC m=+4.558525308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs") pod "network-metrics-daemon-nmn9p" (UID: "80efd7c7-aa08-47ee-a917-251ef8624493") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.371673    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-var-lib-cni-bin\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.371948    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-socket-dir\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.371998    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-system-cni-dir\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.372037    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de993612-482a-4476-a44b-cef040d77485-run\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.372069    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-run-netns\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.372762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.372165    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/473f1449-5a91-400d-8225-88e8716cd98d-multus-daemon-config\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.373570 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.372209    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/473f1449-5a91-400d-8225-88e8716cd98d-host-var-lib-kubelet\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.373570 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.372253    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-run-systemd\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.373570 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.372307    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-var-lib-openvswitch\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.373570 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.372348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-run-ovn-kubernetes\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.373570 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.372449    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70ea03b1-9946-46d6-b68a-9c730d932c72-serviceca\") pod \"node-ca-pxhs2\" (UID: \"70ea03b1-9946-46d6-b68a-9c730d932c72\") " pod="openshift-image-registry/node-ca-pxhs2"
Sep 17 23:10:49.373570 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.372497    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-host-run-netns\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.373570 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.372559    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:49.373570 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.372574    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fe30d8-d67c-4736-86fe-e33d93dafbb0-tmp-dir\") pod \"node-resolver-95nk9\" (UID: \"d0fe30d8-d67c-4736-86fe-e33d93dafbb0\") " pod="openshift-dns/node-resolver-95nk9"
Sep 17 23:10:49.373570 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.372596    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-run-ovn\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.373570 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.372649    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret podName:d48fd517-45d0-445c-836e-7045570e3385 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:49.872633632 +0000 UTC m=+4.559706190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret") pod "global-pull-secret-syncer-s7lg2" (UID: "d48fd517-45d0-445c-836e-7045570e3385") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:49.373570 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.372660    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-device-dir\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.374139 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.374120    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de993612-482a-4476-a44b-cef040d77485-tmp\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.374323 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.374271    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de993612-482a-4476-a44b-cef040d77485-etc-tuned\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.376007 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.375984    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-ovn-node-metrics-cert\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.378534 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.378514    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd8sh\" (UniqueName: \"kubernetes.io/projected/80efd7c7-aa08-47ee-a917-251ef8624493-kube-api-access-hd8sh\") pod \"network-metrics-daemon-nmn9p\" (UID: \"80efd7c7-aa08-47ee-a917-251ef8624493\") " pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:10:49.378606 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.378542    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf6xh\" (UniqueName: \"kubernetes.io/projected/1d723cf3-2dbe-4e15-8327-f0b6dac7801e-kube-api-access-mf6xh\") pod \"aws-ebs-csi-driver-node-84n7w\" (UID: \"1d723cf3-2dbe-4e15-8327-f0b6dac7801e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.378930 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.378910    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb68h\" (UniqueName: \"kubernetes.io/projected/473f1449-5a91-400d-8225-88e8716cd98d-kube-api-access-fb68h\") pod \"multus-t2x98\" (UID: \"473f1449-5a91-400d-8225-88e8716cd98d\") " pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.379545 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.379523    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b41f4c0-9428-4c24-b1bc-725169912218-agent-certs\") pod \"konnectivity-agent-mzjpw\" (UID: \"0b41f4c0-9428-4c24-b1bc-725169912218\") " pod="kube-system/konnectivity-agent-mzjpw"
Sep 17 23:10:49.380105 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.380083    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhczf\" (UniqueName: \"kubernetes.io/projected/70ea03b1-9946-46d6-b68a-9c730d932c72-kube-api-access-zhczf\") pod \"node-ca-pxhs2\" (UID: \"70ea03b1-9946-46d6-b68a-9c730d932c72\") " pod="openshift-image-registry/node-ca-pxhs2"
Sep 17 23:10:49.382199 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.382181    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:49.382296 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.382206    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:49.382296 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.382219    2582 projected.go:196] Error preparing data for projected volume kube-api-access-m9tjl for pod openshift-network-diagnostics/network-check-target-5xnb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:49.382418 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.382306    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl podName:9e2add4f-c624-4d66-a7c5-8136a8ff8943 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:49.882289401 +0000 UTC m=+4.569361963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m9tjl" (UniqueName: "kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl") pod "network-check-target-5xnb7" (UID: "9e2add4f-c624-4d66-a7c5-8136a8ff8943") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:49.386285 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.386233    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xftvw\" (UniqueName: \"kubernetes.io/projected/de993612-482a-4476-a44b-cef040d77485-kube-api-access-xftvw\") pod \"tuned-nbvc8\" (UID: \"de993612-482a-4476-a44b-cef040d77485\") " pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.387696 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.387316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctkqc\" (UniqueName: \"kubernetes.io/projected/70ddc5e6-3ef1-4732-a37e-3fa6d0a84670-kube-api-access-ctkqc\") pod \"ovnkube-node-88d87\" (UID: \"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670\") " pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.387696 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.387655    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njj27\" (UniqueName: \"kubernetes.io/projected/d0fe30d8-d67c-4736-86fe-e33d93dafbb0-kube-api-access-njj27\") pod \"node-resolver-95nk9\" (UID: \"d0fe30d8-d67c-4736-86fe-e33d93dafbb0\") " pod="openshift-dns/node-resolver-95nk9"
Sep 17 23:10:49.389911 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.389891    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5z7r\" (UniqueName: \"kubernetes.io/projected/5b67fba1-4bba-4dc6-b35f-a9e3375ffd79-kube-api-access-x5z7r\") pod \"iptables-alerter-d9hfz\" (UID: \"5b67fba1-4bba-4dc6-b35f-a9e3375ffd79\") " pod="openshift-network-operator/iptables-alerter-d9hfz"
Sep 17 23:10:49.394091 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.394054    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal" event={"ID":"ad31e4fc899beb2f71ca79970426a20a","Type":"ContainerStarted","Data":"5a29c9c01650935f21653ef3f3de7b7c285593568f558f0ff610dc2086a279bd"}
Sep 17 23:10:49.395689 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.395656    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" event={"ID":"8486b3e20df1174f1076e7f4de44e909","Type":"ContainerStarted","Data":"72f69bd49d68a40d4349e782f1cbc5552485cf3ad36ce060f500e4b2ca8d393a"}
Sep 17 23:10:49.472156 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.472081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/720f72ab-3b57-4871-9d92-4f546cf4217f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.472156 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.472131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/720f72ab-3b57-4871-9d92-4f546cf4217f-system-cni-dir\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.472342 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.472158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2v98\" (UniqueName: \"kubernetes.io/projected/720f72ab-3b57-4871-9d92-4f546cf4217f-kube-api-access-w2v98\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.472342 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.472200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/720f72ab-3b57-4871-9d92-4f546cf4217f-cnibin\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.472342 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.472233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/720f72ab-3b57-4871-9d92-4f546cf4217f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.472342 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.472266    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/720f72ab-3b57-4871-9d92-4f546cf4217f-cni-binary-copy\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.472342 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.472282    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/720f72ab-3b57-4871-9d92-4f546cf4217f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.472891 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.472869    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/720f72ab-3b57-4871-9d92-4f546cf4217f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.473004 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.472977    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/720f72ab-3b57-4871-9d92-4f546cf4217f-system-cni-dir\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.473076 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.473018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/720f72ab-3b57-4871-9d92-4f546cf4217f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.473076 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.473031    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/720f72ab-3b57-4871-9d92-4f546cf4217f-cnibin\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.473354 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.473334    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/720f72ab-3b57-4871-9d92-4f546cf4217f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.473723 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.473671    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/720f72ab-3b57-4871-9d92-4f546cf4217f-cni-binary-copy\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.483326 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.483305    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2v98\" (UniqueName: \"kubernetes.io/projected/720f72ab-3b57-4871-9d92-4f546cf4217f-kube-api-access-w2v98\") pod \"multus-additional-cni-plugins-j6vfx\" (UID: \"720f72ab-3b57-4871-9d92-4f546cf4217f\") " pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.544951 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.544910    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mzjpw"
Sep 17 23:10:49.551934 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.551906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pxhs2"
Sep 17 23:10:49.564708 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.564688    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t2x98"
Sep 17 23:10:49.577462 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.577433    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:10:49.585161 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.585141    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w"
Sep 17 23:10:49.592816 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.592794    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nbvc8"
Sep 17 23:10:49.600412 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.600392    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-95nk9"
Sep 17 23:10:49.607899 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.607881    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d9hfz"
Sep 17 23:10:49.614460 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.614442    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j6vfx"
Sep 17 23:10:49.875506 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.875427    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:49.875506 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.875469    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs\") pod \"network-metrics-daemon-nmn9p\" (UID: \"80efd7c7-aa08-47ee-a917-251ef8624493\") " pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:10:49.875692 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.875577    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:49.875692 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.875581    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:49.875692 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.875628    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs podName:80efd7c7-aa08-47ee-a917-251ef8624493 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:50.875613297 +0000 UTC m=+5.562685866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs") pod "network-metrics-daemon-nmn9p" (UID: "80efd7c7-aa08-47ee-a917-251ef8624493") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:49.875692 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.875641    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret podName:d48fd517-45d0-445c-836e-7045570e3385 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:50.87563528 +0000 UTC m=+5.562707834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret") pod "global-pull-secret-syncer-s7lg2" (UID: "d48fd517-45d0-445c-836e-7045570e3385") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:49.976546 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:49.976516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tjl\" (UniqueName: \"kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl\") pod \"network-check-target-5xnb7\" (UID: \"9e2add4f-c624-4d66-a7c5-8136a8ff8943\") " pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:10:49.976701 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.976638    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:49.976701 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.976653    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:49.976701 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.976662    2582 projected.go:196] Error preparing data for projected volume kube-api-access-m9tjl for pod openshift-network-diagnostics/network-check-target-5xnb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:49.976811 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:49.976718    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl podName:9e2add4f-c624-4d66-a7c5-8136a8ff8943 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:50.97670566 +0000 UTC m=+5.663778216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9tjl" (UniqueName: "kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl") pod "network-check-target-5xnb7" (UID: "9e2add4f-c624-4d66-a7c5-8136a8ff8943") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:50.330719 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:50.330681    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:05:49 +0000 UTC" deadline="2028-03-11 21:04:14.976698112 +0000 UTC"
Sep 17 23:10:50.330719 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:50.330712    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12981h53m24.64598968s"
Sep 17 23:10:50.360097 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:50.360071    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ddc5e6_3ef1_4732_a37e_3fa6d0a84670.slice/crio-bf8a1d77078cd23c6766b8969d3ffb01f67b6545ac29df7d7b4433032f3dc582 WatchSource:0}: Error finding container bf8a1d77078cd23c6766b8969d3ffb01f67b6545ac29df7d7b4433032f3dc582: Status 404 returned error can't find the container with id bf8a1d77078cd23c6766b8969d3ffb01f67b6545ac29df7d7b4433032f3dc582
Sep 17 23:10:50.361897 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:50.361753    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473f1449_5a91_400d_8225_88e8716cd98d.slice/crio-704973e30af72b437270f373e8d3883fbd1b4994db47d91e32272b252b75dd15 WatchSource:0}: Error finding container 704973e30af72b437270f373e8d3883fbd1b4994db47d91e32272b252b75dd15: Status 404 returned error can't find the container with id 704973e30af72b437270f373e8d3883fbd1b4994db47d91e32272b252b75dd15
Sep 17 23:10:50.362799 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:50.362633    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fe30d8_d67c_4736_86fe_e33d93dafbb0.slice/crio-2449f91fc9b4878385fa14e2e92ba1e1d797bcdf76fa18d01e18ca59ce77229a WatchSource:0}: Error finding container 2449f91fc9b4878385fa14e2e92ba1e1d797bcdf76fa18d01e18ca59ce77229a: Status 404 returned error can't find the container with id 2449f91fc9b4878385fa14e2e92ba1e1d797bcdf76fa18d01e18ca59ce77229a
Sep 17 23:10:50.364216 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:50.364170    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720f72ab_3b57_4871_9d92_4f546cf4217f.slice/crio-63780a1b327ca7abe1cf9ad84170c3b2546213729941ad9f04ac2708e7ac02d6 WatchSource:0}: Error finding container 63780a1b327ca7abe1cf9ad84170c3b2546213729941ad9f04ac2708e7ac02d6: Status 404 returned error can't find the container with id 63780a1b327ca7abe1cf9ad84170c3b2546213729941ad9f04ac2708e7ac02d6
Sep 17 23:10:50.365011 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:50.364989    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d723cf3_2dbe_4e15_8327_f0b6dac7801e.slice/crio-85635c97227b5aa4510d39cb26ed2db5a7480b11964ff143faabe88dc155bcd6 WatchSource:0}: Error finding container 85635c97227b5aa4510d39cb26ed2db5a7480b11964ff143faabe88dc155bcd6: Status 404 returned error can't find the container with id 85635c97227b5aa4510d39cb26ed2db5a7480b11964ff143faabe88dc155bcd6
Sep 17 23:10:50.365913 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:10:50.365745    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ea03b1_9946_46d6_b68a_9c730d932c72.slice/crio-fbf2c971eced14d585541c00ca88a8c1ae8fcc7e28e6bc8f6a92e237e444d224 WatchSource:0}: Error finding container fbf2c971eced14d585541c00ca88a8c1ae8fcc7e28e6bc8f6a92e237e444d224: Status 404 returned error can't find the container with id fbf2c971eced14d585541c00ca88a8c1ae8fcc7e28e6bc8f6a92e237e444d224
Sep 17 23:10:50.402077 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:50.402050    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t2x98" event={"ID":"473f1449-5a91-400d-8225-88e8716cd98d","Type":"ContainerStarted","Data":"704973e30af72b437270f373e8d3883fbd1b4994db47d91e32272b252b75dd15"}
Sep 17 23:10:50.403739 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:50.403717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nbvc8" event={"ID":"de993612-482a-4476-a44b-cef040d77485","Type":"ContainerStarted","Data":"f82c1d528cb576b8cfef26485482f947c39506c4d264c1252d0eabc24b624511"}
Sep 17 23:10:50.404931 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:50.404908    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mzjpw" event={"ID":"0b41f4c0-9428-4c24-b1bc-725169912218","Type":"ContainerStarted","Data":"b65f07a63e4cd8a2c3557119f36db2b0a9143f4121d5fab0ec354b237d524400"}
Sep 17 23:10:50.405970 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:50.405949    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pxhs2" event={"ID":"70ea03b1-9946-46d6-b68a-9c730d932c72","Type":"ContainerStarted","Data":"fbf2c971eced14d585541c00ca88a8c1ae8fcc7e28e6bc8f6a92e237e444d224"}
Sep 17 23:10:50.407073 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:50.407054    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6vfx" event={"ID":"720f72ab-3b57-4871-9d92-4f546cf4217f","Type":"ContainerStarted","Data":"63780a1b327ca7abe1cf9ad84170c3b2546213729941ad9f04ac2708e7ac02d6"}
Sep 17 23:10:50.409538 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:50.409518    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88d87" event={"ID":"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670","Type":"ContainerStarted","Data":"bf8a1d77078cd23c6766b8969d3ffb01f67b6545ac29df7d7b4433032f3dc582"}
Sep 17 23:10:50.411930 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:50.411905    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d9hfz" event={"ID":"5b67fba1-4bba-4dc6-b35f-a9e3375ffd79","Type":"ContainerStarted","Data":"94e10a60f1eade7186339adc35fc47b85dc535d494e6f69a575628fca942dc7a"}
Sep 17 23:10:50.414099 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:50.414073    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w" event={"ID":"1d723cf3-2dbe-4e15-8327-f0b6dac7801e","Type":"ContainerStarted","Data":"85635c97227b5aa4510d39cb26ed2db5a7480b11964ff143faabe88dc155bcd6"}
Sep 17 23:10:50.415252 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:50.415227    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-95nk9" event={"ID":"d0fe30d8-d67c-4736-86fe-e33d93dafbb0","Type":"ContainerStarted","Data":"2449f91fc9b4878385fa14e2e92ba1e1d797bcdf76fa18d01e18ca59ce77229a"}
Sep 17 23:10:50.884348 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:50.884103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:50.884524 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:50.884376    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs\") pod \"network-metrics-daemon-nmn9p\" (UID: \"80efd7c7-aa08-47ee-a917-251ef8624493\") " pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:10:50.884524 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:50.884507    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:50.884624 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:50.884567    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs podName:80efd7c7-aa08-47ee-a917-251ef8624493 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:52.884548477 +0000 UTC m=+7.571621039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs") pod "network-metrics-daemon-nmn9p" (UID: "80efd7c7-aa08-47ee-a917-251ef8624493") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:50.884989 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:50.884968    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:50.885072 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:50.885025    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret podName:d48fd517-45d0-445c-836e-7045570e3385 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:52.885009979 +0000 UTC m=+7.572082540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret") pod "global-pull-secret-syncer-s7lg2" (UID: "d48fd517-45d0-445c-836e-7045570e3385") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:50.985245 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:50.985211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tjl\" (UniqueName: \"kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl\") pod \"network-check-target-5xnb7\" (UID: \"9e2add4f-c624-4d66-a7c5-8136a8ff8943\") " pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:10:50.985408 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:50.985373    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:50.985408 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:50.985393    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:50.985408 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:50.985406    2582 projected.go:196] Error preparing data for projected volume kube-api-access-m9tjl for pod openshift-network-diagnostics/network-check-target-5xnb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:50.985578 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:50.985459    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl podName:9e2add4f-c624-4d66-a7c5-8136a8ff8943 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:52.985442009 +0000 UTC m=+7.672514569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9tjl" (UniqueName: "kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl") pod "network-check-target-5xnb7" (UID: "9e2add4f-c624-4d66-a7c5-8136a8ff8943") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:51.331322 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:51.331226    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:05:49 +0000 UTC" deadline="2028-04-14 01:21:24.890779488 +0000 UTC"
Sep 17 23:10:51.331322 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:51.331267    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13778h10m33.559516567s"
Sep 17 23:10:51.391603 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:51.391469    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:51.391603 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:51.391583    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:10:51.392140 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:51.391966    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:10:51.392140 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:51.392055    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:10:51.392140 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:51.392118    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:10:51.392329 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:51.392195    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:10:51.437733 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:51.437695    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal" event={"ID":"ad31e4fc899beb2f71ca79970426a20a","Type":"ContainerStarted","Data":"68ad906ae24bd4e09fa6b691f4304d7cbff55a66cf430e61522218b3b60e7524"}
Sep 17 23:10:51.455282 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:51.455251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" event={"ID":"8486b3e20df1174f1076e7f4de44e909","Type":"ContainerDied","Data":"fc06a873ad549101e49652e34aa148ed8528b30103d5af186208ef71cfdb2aa9"}
Sep 17 23:10:51.455788 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:51.455749    2582 generic.go:356] "Generic (PLEG): container finished" podID="8486b3e20df1174f1076e7f4de44e909" containerID="fc06a873ad549101e49652e34aa148ed8528b30103d5af186208ef71cfdb2aa9" exitCode=0
Sep 17 23:10:51.479586 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:51.475310    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-43.ec2.internal" podStartSLOduration=3.475291808 podStartE2EDuration="3.475291808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:51.458970408 +0000 UTC m=+6.146042987" watchObservedRunningTime="2026-09-17 23:10:51.475291808 +0000 UTC m=+6.162364385"
Sep 17 23:10:52.473466 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:52.473390    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" event={"ID":"8486b3e20df1174f1076e7f4de44e909","Type":"ContainerStarted","Data":"2aa81710314fcd5aaed2baad767b55bdd8a2a65f6808158e0c025b94b44c267e"}
Sep 17 23:10:52.491781 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:52.490816    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-43.ec2.internal" podStartSLOduration=4.490795086 podStartE2EDuration="4.490795086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:52.490389829 +0000 UTC m=+7.177462414" watchObservedRunningTime="2026-09-17 23:10:52.490795086 +0000 UTC m=+7.177867671"
Sep 17 23:10:52.902440 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:52.902404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:52.902637 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:52.902463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs\") pod \"network-metrics-daemon-nmn9p\" (UID: \"80efd7c7-aa08-47ee-a917-251ef8624493\") " pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:10:52.902637 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:52.902624    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:52.902768 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:52.902682    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs podName:80efd7c7-aa08-47ee-a917-251ef8624493 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:56.902665305 +0000 UTC m=+11.589737865 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs") pod "network-metrics-daemon-nmn9p" (UID: "80efd7c7-aa08-47ee-a917-251ef8624493") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:52.903108 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:52.902993    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:52.903108 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:52.903072    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret podName:d48fd517-45d0-445c-836e-7045570e3385 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:56.90305474 +0000 UTC m=+11.590127300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret") pod "global-pull-secret-syncer-s7lg2" (UID: "d48fd517-45d0-445c-836e-7045570e3385") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:53.002859 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:53.002768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tjl\" (UniqueName: \"kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl\") pod \"network-check-target-5xnb7\" (UID: \"9e2add4f-c624-4d66-a7c5-8136a8ff8943\") " pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:10:53.003108 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:53.002987    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:53.003108 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:53.003009    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:53.003108 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:53.003022    2582 projected.go:196] Error preparing data for projected volume kube-api-access-m9tjl for pod openshift-network-diagnostics/network-check-target-5xnb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:53.003108 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:53.003073    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl podName:9e2add4f-c624-4d66-a7c5-8136a8ff8943 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:57.003058048 +0000 UTC m=+11.690130609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9tjl" (UniqueName: "kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl") pod "network-check-target-5xnb7" (UID: "9e2add4f-c624-4d66-a7c5-8136a8ff8943") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:53.389661 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:53.388951    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:53.389661 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:53.389086    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:10:53.389661 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:53.389482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:10:53.389661 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:53.389583    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:10:53.390713 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:53.390393    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:10:53.390713 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:53.390634    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:10:55.388984 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:55.388952    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:10:55.388984 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:55.388988    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:55.389561 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:55.388968    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:10:55.389561 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:55.389098    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:10:55.389561 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:55.389182    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:10:55.389561 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:55.389242    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:10:56.937046 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:56.937007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:56.937514 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:56.937064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs\") pod \"network-metrics-daemon-nmn9p\" (UID: \"80efd7c7-aa08-47ee-a917-251ef8624493\") " pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:10:56.937514 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:56.937182    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:56.937514 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:56.937208    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:56.937514 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:56.937261    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs podName:80efd7c7-aa08-47ee-a917-251ef8624493 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:04.937242822 +0000 UTC m=+19.624315400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs") pod "network-metrics-daemon-nmn9p" (UID: "80efd7c7-aa08-47ee-a917-251ef8624493") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:56.937514 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:56.937279    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret podName:d48fd517-45d0-445c-836e-7045570e3385 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:04.937270026 +0000 UTC m=+19.624342587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret") pod "global-pull-secret-syncer-s7lg2" (UID: "d48fd517-45d0-445c-836e-7045570e3385") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:57.038291 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:57.038253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tjl\" (UniqueName: \"kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl\") pod \"network-check-target-5xnb7\" (UID: \"9e2add4f-c624-4d66-a7c5-8136a8ff8943\") " pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:10:57.038483 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:57.038394    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:57.038483 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:57.038415    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:57.038483 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:57.038427    2582 projected.go:196] Error preparing data for projected volume kube-api-access-m9tjl for pod openshift-network-diagnostics/network-check-target-5xnb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:57.038483 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:57.038484    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl podName:9e2add4f-c624-4d66-a7c5-8136a8ff8943 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:05.03846627 +0000 UTC m=+19.725538846 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9tjl" (UniqueName: "kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl") pod "network-check-target-5xnb7" (UID: "9e2add4f-c624-4d66-a7c5-8136a8ff8943") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:57.392510 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:57.390731    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:10:57.392510 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:57.390868    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:10:57.392510 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:57.391220    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:10:57.392510 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:57.391317    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:10:57.392510 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:57.391368    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:57.392510 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:57.391435    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:10:59.389198 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:59.389164    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:10:59.389198 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:59.389196    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:10:59.389708 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:10:59.389259    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:10:59.389708 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:59.389379    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:10:59.389708 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:59.389497    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:10:59.389708 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:10:59.389576    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:11:01.388828 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:01.388794    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:01.389260 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:01.388857    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:01.389260 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:01.388794    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:01.389260 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:01.388938    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:11:01.389260 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:01.389030    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:11:01.389260 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:01.389123    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:11:03.388787 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:03.388751    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:03.389277 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:03.388751    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:03.389277 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:03.388912    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:11:03.389277 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:03.388930    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:11:03.389277 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:03.388757    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:03.389277 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:03.389009    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:11:04.998778 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:04.998737    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs\") pod \"network-metrics-daemon-nmn9p\" (UID: \"80efd7c7-aa08-47ee-a917-251ef8624493\") " pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:04.999270 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:04.998869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:04.999270 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:04.998921    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:04.999270 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:04.998972    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:04.999270 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:04.998997    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs podName:80efd7c7-aa08-47ee-a917-251ef8624493 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:20.9989801 +0000 UTC m=+35.686052659 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs") pod "network-metrics-daemon-nmn9p" (UID: "80efd7c7-aa08-47ee-a917-251ef8624493") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:04.999270 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:04.999023    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret podName:d48fd517-45d0-445c-836e-7045570e3385 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:20.999005641 +0000 UTC m=+35.686078210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret") pod "global-pull-secret-syncer-s7lg2" (UID: "d48fd517-45d0-445c-836e-7045570e3385") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:05.099978 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:05.099942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tjl\" (UniqueName: \"kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl\") pod \"network-check-target-5xnb7\" (UID: \"9e2add4f-c624-4d66-a7c5-8136a8ff8943\") " pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:05.100149 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:05.100133    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:05.100220 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:05.100161    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:05.100220 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:05.100175    2582 projected.go:196] Error preparing data for projected volume kube-api-access-m9tjl for pod openshift-network-diagnostics/network-check-target-5xnb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:05.100340 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:05.100235    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl podName:9e2add4f-c624-4d66-a7c5-8136a8ff8943 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:21.100221803 +0000 UTC m=+35.787294357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9tjl" (UniqueName: "kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl") pod "network-check-target-5xnb7" (UID: "9e2add4f-c624-4d66-a7c5-8136a8ff8943") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:05.388916 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:05.388879    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:05.389070 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:05.388880    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:05.389070 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:05.389011    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:11:05.389180 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:05.389130    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:11:05.389180 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:05.388987    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:05.389305 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:05.389274    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:11:07.390076 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:07.390036    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:07.390539 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:07.390164    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:07.390539 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:07.390271    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:11:07.390539 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:07.390331    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:11:07.390539 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:07.390337    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:07.390539 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:07.390424    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:11:08.530044 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:08.529554    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mzjpw" event={"ID":"0b41f4c0-9428-4c24-b1bc-725169912218","Type":"ContainerStarted","Data":"db60297cd4e3bd72cb42fd91a1699345903cd4a9380d2c05f7fe9679af7b86c7"}
Sep 17 23:11:08.536434 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:08.536393    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6vfx" event={"ID":"720f72ab-3b57-4871-9d92-4f546cf4217f","Type":"ContainerStarted","Data":"6ad383575ae6ce3b80897646ad136443de8f9bdbf5248fe5ff5d0f8bfd4c1448"}
Sep 17 23:11:08.541894 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:08.541655    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-95nk9" event={"ID":"d0fe30d8-d67c-4736-86fe-e33d93dafbb0","Type":"ContainerStarted","Data":"f7fc552d817c0bb222fc468740b7d921d7bdf72e31ef9e9342e17bece5ea084b"}
Sep 17 23:11:08.566433 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:08.565989    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mzjpw" podStartSLOduration=7.653658801 podStartE2EDuration="21.565972471s" totalImagesPullingTime="13.91231367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:08.565964317 +0000 UTC m=+23.253036906" watchObservedRunningTime="2026-09-17 23:11:08.565972471 +0000 UTC m=+23.253045048"
Sep 17 23:11:08.640857 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:08.640747    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-t2x98" podStartSLOduration=3.711532723 podStartE2EDuration="21.640729266s" totalImagesPullingTime="17.929196543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:08.596970565 +0000 UTC m=+23.284043143" watchObservedRunningTime="2026-09-17 23:11:08.640729266 +0000 UTC m=+23.327801847"
Sep 17 23:11:08.641724 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:08.641266    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-95nk9" podStartSLOduration=3.781033201 podStartE2EDuration="21.641251018s" totalImagesPullingTime="17.860217817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:08.634198527 +0000 UTC m=+23.321271103" watchObservedRunningTime="2026-09-17 23:11:08.641251018 +0000 UTC m=+23.328323598"
Sep 17 23:11:09.342634 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.342483    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 23:11:09.391581 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.391553    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:09.391762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.391561    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:09.391762 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:09.391658    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:11:09.391762 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:09.391722    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:11:09.391762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.391744    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:09.391988 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:09.391819    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:11:09.547346 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.547255    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t2x98" event={"ID":"473f1449-5a91-400d-8225-88e8716cd98d","Type":"ContainerStarted","Data":"6b5439dae8ecbb2d820e48935fbfa3043318adb16f68b1e145e18c377b6dcabe"}
Sep 17 23:11:09.548736 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.548711    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nbvc8" event={"ID":"de993612-482a-4476-a44b-cef040d77485","Type":"ContainerStarted","Data":"31379784b340cf34271536ea10d520dfc1942e75efd554616387fbd5dd73dd9a"}
Sep 17 23:11:09.550071 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.550051    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pxhs2" event={"ID":"70ea03b1-9946-46d6-b68a-9c730d932c72","Type":"ContainerStarted","Data":"acd631c2eb8714afd62cf0b3305da5ff12ea3f3b0dd3cf7d7b735e3a4566c024"}
Sep 17 23:11:09.551442 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.551420    2582 generic.go:356] "Generic (PLEG): container finished" podID="720f72ab-3b57-4871-9d92-4f546cf4217f" containerID="6ad383575ae6ce3b80897646ad136443de8f9bdbf5248fe5ff5d0f8bfd4c1448" exitCode=0
Sep 17 23:11:09.551542 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.551486    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6vfx" event={"ID":"720f72ab-3b57-4871-9d92-4f546cf4217f","Type":"ContainerDied","Data":"6ad383575ae6ce3b80897646ad136443de8f9bdbf5248fe5ff5d0f8bfd4c1448"}
Sep 17 23:11:09.554867 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.554823    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88d87" event={"ID":"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670","Type":"ContainerStarted","Data":"b7bf148fdbcbbc44f20e2bb33e43a0468f560afa8f9ff586c55a9f04a60a8b58"}
Sep 17 23:11:09.554948 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.554925    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88d87" event={"ID":"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670","Type":"ContainerStarted","Data":"7338f77ee4551fe0548bae4a050719d7602c9887754ece0e7de91261fc55ed80"}
Sep 17 23:11:09.554989 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.554977    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88d87" event={"ID":"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670","Type":"ContainerStarted","Data":"e24d7566c2b69c5d7122ea07c9707624fb06316a847a1c524373094626598207"}
Sep 17 23:11:09.555040 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.555027    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88d87" event={"ID":"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670","Type":"ContainerStarted","Data":"a600a0141699783a76ed0322d528745dbdd8796039be8045ca36b70392f1c773"}
Sep 17 23:11:09.555091 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.555063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88d87" event={"ID":"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670","Type":"ContainerStarted","Data":"7a1f3af9b520da57d95f3955ec89ced7d91eb4a4dc5984c0adf3a1e3ec6ad628"}
Sep 17 23:11:09.555146 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.555097    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88d87" event={"ID":"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670","Type":"ContainerStarted","Data":"2c1d1ff7fa4e49e1c3aa029022f10838fd07d6bb7b23738da29afc9b30196ede"}
Sep 17 23:11:09.556554 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.556535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w" event={"ID":"1d723cf3-2dbe-4e15-8327-f0b6dac7801e","Type":"ContainerStarted","Data":"50563b0c55eb3220748c99777a92dd2044fca3b5f9653307810ab13e106b0862"}
Sep 17 23:11:09.556643 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.556575    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w" event={"ID":"1d723cf3-2dbe-4e15-8327-f0b6dac7801e","Type":"ContainerStarted","Data":"3c6979909ba5ad3f09711d95b8bcb566e86545ee36335438fa02506d23d65132"}
Sep 17 23:11:09.595563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.595506    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nbvc8" podStartSLOduration=4.696969413 podStartE2EDuration="22.595488056s" totalImagesPullingTime="17.898518643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:09.572443259 +0000 UTC m=+24.259515837" watchObservedRunningTime="2026-09-17 23:11:09.595488056 +0000 UTC m=+24.282560677"
Sep 17 23:11:09.627274 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:09.627218    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pxhs2" podStartSLOduration=4.726157149 podStartE2EDuration="22.627203903s" totalImagesPullingTime="17.901046754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:09.596768271 +0000 UTC m=+24.283840847" watchObservedRunningTime="2026-09-17 23:11:09.627203903 +0000 UTC m=+24.314276531"
Sep 17 23:11:10.333867 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:10.333632    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T23:11:09.342630748Z","UUID":"ee6fdd45-161f-46f5-84ac-039538a9fd3e","Handler":null,"Name":"","Endpoint":""}
Sep 17 23:11:10.335654 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:10.335625    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 23:11:10.335792 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:10.335666    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 23:11:10.561249 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:10.561208    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d9hfz" event={"ID":"5b67fba1-4bba-4dc6-b35f-a9e3375ffd79","Type":"ContainerStarted","Data":"1e12298c4808407e5044b2cec4527e750bd765bf4094d3e6ac4983cc41739d10"}
Sep 17 23:11:10.563818 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:10.563741    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w" event={"ID":"1d723cf3-2dbe-4e15-8327-f0b6dac7801e","Type":"ContainerStarted","Data":"8695807d7a4e16371bde7b0b6fd1028d16369a74f3bde5e1df91330137ce31f7"}
Sep 17 23:11:10.620252 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:10.620149    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d9hfz" podStartSLOduration=6.113467746 podStartE2EDuration="23.620126887s" totalImagesPullingTime="17.506659141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:10.584474519 +0000 UTC m=+25.271547097" watchObservedRunningTime="2026-09-17 23:11:10.620126887 +0000 UTC m=+25.307199502"
Sep 17 23:11:11.191867 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:11.191807    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mzjpw"
Sep 17 23:11:11.192513 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:11.192488    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mzjpw"
Sep 17 23:11:11.214986 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:11.214937    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-84n7w" podStartSLOduration=4.91885075 podStartE2EDuration="24.214920309s" totalImagesPullingTime="19.296069559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:10.619969933 +0000 UTC m=+25.307042510" watchObservedRunningTime="2026-09-17 23:11:11.214920309 +0000 UTC m=+25.901992908"
Sep 17 23:11:11.390634 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:11.390600    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:11.390871 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:11.390711    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:11:11.391067 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:11.391052    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:11.391146 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:11.391122    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:11:11.391198 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:11.391179    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:11.391235 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:11.391227    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:11:11.570597 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:11.570554    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88d87" event={"ID":"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670","Type":"ContainerStarted","Data":"9ee1602cc3de1c3bf1e3bf6395272e5356a82921140c7debfb987413b821194b"}
Sep 17 23:11:11.571071 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:11.570939    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mzjpw"
Sep 17 23:11:11.571592 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:11.571566    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mzjpw"
Sep 17 23:11:13.390328 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:13.390299    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:13.390791 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:13.390428    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:11:13.390791 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:13.390482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:13.390791 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:13.390529    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:11:13.390978 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:13.390852    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:13.390978 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:13.390910    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:11:14.578923 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:14.578693    2582 generic.go:356] "Generic (PLEG): container finished" podID="720f72ab-3b57-4871-9d92-4f546cf4217f" containerID="15c1096901760a58928c159f30f4ebc571992ef5385959deedc7beb18167bc2c" exitCode=0
Sep 17 23:11:14.579770 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:14.578779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6vfx" event={"ID":"720f72ab-3b57-4871-9d92-4f546cf4217f","Type":"ContainerDied","Data":"15c1096901760a58928c159f30f4ebc571992ef5385959deedc7beb18167bc2c"}
Sep 17 23:11:14.583198 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:14.583174    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88d87" event={"ID":"70ddc5e6-3ef1-4732-a37e-3fa6d0a84670","Type":"ContainerStarted","Data":"de4b7560998485958129dc902303413d8b096b4a9db6d55505cbbfa61809a624"}
Sep 17 23:11:14.583459 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:14.583444    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:11:14.583522 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:14.583496    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:11:14.600672 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:14.600641    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:11:14.650261 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:14.650207    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-88d87" podStartSLOduration=9.693698203 podStartE2EDuration="27.650192621s" totalImagesPullingTime="17.956494418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:14.6500706 +0000 UTC m=+29.337143177" watchObservedRunningTime="2026-09-17 23:11:14.650192621 +0000 UTC m=+29.337265198"
Sep 17 23:11:15.388652 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:15.388391    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:15.388823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:15.388399    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:15.388823 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:15.388413    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:15.388823 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:15.388775    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:11:15.388973 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:15.388890    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:11:15.389012 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:15.388975    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:11:15.586168 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:15.586134    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:11:15.604222 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:15.604194    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:11:15.629101 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:15.628989    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s7lg2"]
Sep 17 23:11:15.629249 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:15.629118    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:15.629249 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:15.629229    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:11:15.629789 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:15.629756    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nmn9p"]
Sep 17 23:11:15.629949 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:15.629880    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:15.630016 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:15.629980    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:11:15.630812 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:15.630790    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5xnb7"]
Sep 17 23:11:15.630924 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:15.630889    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:15.630979 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:15.630963    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:11:16.590470 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:16.590433    2582 generic.go:356] "Generic (PLEG): container finished" podID="720f72ab-3b57-4871-9d92-4f546cf4217f" containerID="f60fa4567b84c50545fbea39230e10127fe6647dd63832f968d2d572c65a1773" exitCode=0
Sep 17 23:11:16.591049 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:16.590549    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6vfx" event={"ID":"720f72ab-3b57-4871-9d92-4f546cf4217f","Type":"ContainerDied","Data":"f60fa4567b84c50545fbea39230e10127fe6647dd63832f968d2d572c65a1773"}
Sep 17 23:11:17.389727 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:17.389692    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:17.389918 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:17.389771    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:17.389918 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:17.389783    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:11:17.389918 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:17.389905    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:11:17.390033 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:17.389938    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:17.390033 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:17.390015    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:11:18.597743 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:18.597708    2582 generic.go:356] "Generic (PLEG): container finished" podID="720f72ab-3b57-4871-9d92-4f546cf4217f" containerID="709a50e2c0b230454bcf84901dfb1dc9acefb9f5298a1a77c0564e4b51551863" exitCode=0
Sep 17 23:11:18.598142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:18.597766    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6vfx" event={"ID":"720f72ab-3b57-4871-9d92-4f546cf4217f","Type":"ContainerDied","Data":"709a50e2c0b230454bcf84901dfb1dc9acefb9f5298a1a77c0564e4b51551863"}
Sep 17 23:11:19.388573 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:19.388484    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:19.388573 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:19.388516    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:19.392581 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:19.389319    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:19.392581 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:19.389469    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5xnb7" podUID="9e2add4f-c624-4d66-a7c5-8136a8ff8943"
Sep 17 23:11:19.392581 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:19.389491    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmn9p" podUID="80efd7c7-aa08-47ee-a917-251ef8624493"
Sep 17 23:11:19.392581 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:19.389658    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s7lg2" podUID="d48fd517-45d0-445c-836e-7045570e3385"
Sep 17 23:11:21.020167 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.019917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:21.020631 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.020199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs\") pod \"network-metrics-daemon-nmn9p\" (UID: \"80efd7c7-aa08-47ee-a917-251ef8624493\") " pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:21.020631 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:21.020079    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:21.020631 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:21.020316    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret podName:d48fd517-45d0-445c-836e-7045570e3385 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:53.020291497 +0000 UTC m=+67.707364075 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret") pod "global-pull-secret-syncer-s7lg2" (UID: "d48fd517-45d0-445c-836e-7045570e3385") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:21.020631 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:21.020351    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:21.020631 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:21.020417    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs podName:80efd7c7-aa08-47ee-a917-251ef8624493 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:53.020401478 +0000 UTC m=+67.707474053 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs") pod "network-metrics-daemon-nmn9p" (UID: "80efd7c7-aa08-47ee-a917-251ef8624493") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:21.121047 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.120958    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tjl\" (UniqueName: \"kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl\") pod \"network-check-target-5xnb7\" (UID: \"9e2add4f-c624-4d66-a7c5-8136a8ff8943\") " pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:21.121217 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:21.121146    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:21.121217 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:21.121171    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:21.121217 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:21.121184    2582 projected.go:196] Error preparing data for projected volume kube-api-access-m9tjl for pod openshift-network-diagnostics/network-check-target-5xnb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:21.121361 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:21.121251    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl podName:9e2add4f-c624-4d66-a7c5-8136a8ff8943 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:53.121230396 +0000 UTC m=+67.808302967 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m9tjl" (UniqueName: "kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl") pod "network-check-target-5xnb7" (UID: "9e2add4f-c624-4d66-a7c5-8136a8ff8943") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:21.173104 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.173081    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-43.ec2.internal" event="NodeReady"
Sep 17 23:11:21.173235 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.173206    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 23:11:21.236105 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.236074    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-st4l9"]
Sep 17 23:11:21.253904 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.253865    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-msg6l"]
Sep 17 23:11:21.254095 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.253980    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-st4l9"
Sep 17 23:11:21.257032 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.256749    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 23:11:21.257032 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.256775    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 23:11:21.257032 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.256812    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 23:11:21.257224 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.257080    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nkbz8\""
Sep 17 23:11:21.268412 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.268377    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-st4l9"]
Sep 17 23:11:21.268552 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.268421    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-msg6l"]
Sep 17 23:11:21.268552 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.268509    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:21.271252 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.271232    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-42llv\""
Sep 17 23:11:21.271541 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.271525    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 23:11:21.272093 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.272080    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 23:11:21.323263 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.323222    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkddw\" (UniqueName: \"kubernetes.io/projected/27fd2159-4f7f-477e-89f6-9fb97acf078a-kube-api-access-mkddw\") pod \"ingress-canary-st4l9\" (UID: \"27fd2159-4f7f-477e-89f6-9fb97acf078a\") " pod="openshift-ingress-canary/ingress-canary-st4l9"
Sep 17 23:11:21.323469 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.323352    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27fd2159-4f7f-477e-89f6-9fb97acf078a-cert\") pod \"ingress-canary-st4l9\" (UID: \"27fd2159-4f7f-477e-89f6-9fb97acf078a\") " pod="openshift-ingress-canary/ingress-canary-st4l9"
Sep 17 23:11:21.388563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.388532    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:21.388751 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.388605    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:21.388751 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.388720    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:21.392828 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.392802    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:11:21.393127 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.393105    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:11:21.393127 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.393114    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jknmn\""
Sep 17 23:11:21.393278 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.393167    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k7845\""
Sep 17 23:11:21.393406 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.393389    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:11:21.393603 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.393576    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:11:21.424064 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.424032    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27fd2159-4f7f-477e-89f6-9fb97acf078a-cert\") pod \"ingress-canary-st4l9\" (UID: \"27fd2159-4f7f-477e-89f6-9fb97acf078a\") " pod="openshift-ingress-canary/ingress-canary-st4l9"
Sep 17 23:11:21.424231 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.424111    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:21.424231 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.424182    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkddw\" (UniqueName: \"kubernetes.io/projected/27fd2159-4f7f-477e-89f6-9fb97acf078a-kube-api-access-mkddw\") pod \"ingress-canary-st4l9\" (UID: \"27fd2159-4f7f-477e-89f6-9fb97acf078a\") " pod="openshift-ingress-canary/ingress-canary-st4l9"
Sep 17 23:11:21.424343 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.424206    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a14c19c4-990c-4ec5-996c-bbaca8aafc93-config-volume\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:21.424343 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.424294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14c19c4-990c-4ec5-996c-bbaca8aafc93-tmp-dir\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:21.424343 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.424318    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjfvd\" (UniqueName: \"kubernetes.io/projected/a14c19c4-990c-4ec5-996c-bbaca8aafc93-kube-api-access-zjfvd\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:21.429255 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.429228    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27fd2159-4f7f-477e-89f6-9fb97acf078a-cert\") pod \"ingress-canary-st4l9\" (UID: \"27fd2159-4f7f-477e-89f6-9fb97acf078a\") " pod="openshift-ingress-canary/ingress-canary-st4l9"
Sep 17 23:11:21.438753 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.438724    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkddw\" (UniqueName: \"kubernetes.io/projected/27fd2159-4f7f-477e-89f6-9fb97acf078a-kube-api-access-mkddw\") pod \"ingress-canary-st4l9\" (UID: \"27fd2159-4f7f-477e-89f6-9fb97acf078a\") " pod="openshift-ingress-canary/ingress-canary-st4l9"
Sep 17 23:11:21.525347 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.525310    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:21.525524 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.525389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a14c19c4-990c-4ec5-996c-bbaca8aafc93-config-volume\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:21.525524 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.525432    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14c19c4-990c-4ec5-996c-bbaca8aafc93-tmp-dir\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:21.525524 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.525456    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjfvd\" (UniqueName: \"kubernetes.io/projected/a14c19c4-990c-4ec5-996c-bbaca8aafc93-kube-api-access-zjfvd\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:21.525524 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:21.525499    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:21.525720 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:21.525565    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls podName:a14c19c4-990c-4ec5-996c-bbaca8aafc93 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:22.02554895 +0000 UTC m=+36.712621508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls") pod "dns-default-msg6l" (UID: "a14c19c4-990c-4ec5-996c-bbaca8aafc93") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:21.525942 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.525925    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14c19c4-990c-4ec5-996c-bbaca8aafc93-tmp-dir\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:21.526071 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.526053    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a14c19c4-990c-4ec5-996c-bbaca8aafc93-config-volume\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:21.537577 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.537543    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjfvd\" (UniqueName: \"kubernetes.io/projected/a14c19c4-990c-4ec5-996c-bbaca8aafc93-kube-api-access-zjfvd\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:21.564530 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:21.564495    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-st4l9"
Sep 17 23:11:22.028988 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:22.028946    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:22.029504 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:22.029118    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:22.029504 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:22.029189    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls podName:a14c19c4-990c-4ec5-996c-bbaca8aafc93 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:23.029169558 +0000 UTC m=+37.716242133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls") pod "dns-default-msg6l" (UID: "a14c19c4-990c-4ec5-996c-bbaca8aafc93") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:23.036257 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:23.036214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:23.036890 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:23.036361    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:23.036890 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:23.036422    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls podName:a14c19c4-990c-4ec5-996c-bbaca8aafc93 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:25.036406788 +0000 UTC m=+39.723479346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls") pod "dns-default-msg6l" (UID: "a14c19c4-990c-4ec5-996c-bbaca8aafc93") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:23.127746 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:23.127605    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-st4l9"]
Sep 17 23:11:23.159117 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:11:23.159078    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fd2159_4f7f_477e_89f6_9fb97acf078a.slice/crio-3d96afe6854f498105926468cb7bb8fcc8e0d77dbbc55ddbd40945c0738811d7 WatchSource:0}: Error finding container 3d96afe6854f498105926468cb7bb8fcc8e0d77dbbc55ddbd40945c0738811d7: Status 404 returned error can't find the container with id 3d96afe6854f498105926468cb7bb8fcc8e0d77dbbc55ddbd40945c0738811d7
Sep 17 23:11:23.614865 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:23.614277    2582 generic.go:356] "Generic (PLEG): container finished" podID="720f72ab-3b57-4871-9d92-4f546cf4217f" containerID="cd8ef134de5acd561ce6d7c610ade5d20e7051893a4d25fe5281fa59177cf729" exitCode=0
Sep 17 23:11:23.614865 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:23.614373    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6vfx" event={"ID":"720f72ab-3b57-4871-9d92-4f546cf4217f","Type":"ContainerDied","Data":"cd8ef134de5acd561ce6d7c610ade5d20e7051893a4d25fe5281fa59177cf729"}
Sep 17 23:11:23.616235 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:23.616202    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-st4l9" event={"ID":"27fd2159-4f7f-477e-89f6-9fb97acf078a","Type":"ContainerStarted","Data":"3d96afe6854f498105926468cb7bb8fcc8e0d77dbbc55ddbd40945c0738811d7"}
Sep 17 23:11:24.622969 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:24.622921    2582 generic.go:356] "Generic (PLEG): container finished" podID="720f72ab-3b57-4871-9d92-4f546cf4217f" containerID="f8b3e277497f2eba1020e4cbba271ad657373b2473664906acbcbd007e88406c" exitCode=0
Sep 17 23:11:24.623403 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:24.622987    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6vfx" event={"ID":"720f72ab-3b57-4871-9d92-4f546cf4217f","Type":"ContainerDied","Data":"f8b3e277497f2eba1020e4cbba271ad657373b2473664906acbcbd007e88406c"}
Sep 17 23:11:25.052758 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:25.052719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:25.052960 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:25.052910    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:25.053008 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:25.052994    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls podName:a14c19c4-990c-4ec5-996c-bbaca8aafc93 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:29.052977683 +0000 UTC m=+43.740050254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls") pod "dns-default-msg6l" (UID: "a14c19c4-990c-4ec5-996c-bbaca8aafc93") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:25.629646 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:25.629563    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j6vfx" event={"ID":"720f72ab-3b57-4871-9d92-4f546cf4217f","Type":"ContainerStarted","Data":"76ab977fef609cbbb5dcbe500e1aa86fefd31e2e649499ff24f7529ecacaed00"}
Sep 17 23:11:25.663977 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:25.663929    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j6vfx" podStartSLOduration=9.876617123 podStartE2EDuration="38.663913718s" totalImagesPullingTime="28.787296595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:25.663667723 +0000 UTC m=+40.350740309" watchObservedRunningTime="2026-09-17 23:11:25.663913718 +0000 UTC m=+40.350986295"
Sep 17 23:11:26.633945 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:26.633908    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-st4l9" event={"ID":"27fd2159-4f7f-477e-89f6-9fb97acf078a","Type":"ContainerStarted","Data":"1558bace245f64eb7b1c03eb96d23532d8cfcadf39a87cc10aa039c1b28da6d0"}
Sep 17 23:11:26.653027 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:26.652973    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-st4l9" podStartSLOduration=3.169622613 podStartE2EDuration="5.652955617s" totalImagesPullingTime="2.483333004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:26.652521615 +0000 UTC m=+41.339594205" watchObservedRunningTime="2026-09-17 23:11:26.652955617 +0000 UTC m=+41.340028193"
Sep 17 23:11:29.081086 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:29.081031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:29.081555 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:29.081174    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:29.081555 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:29.081230    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls podName:a14c19c4-990c-4ec5-996c-bbaca8aafc93 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:37.0812151 +0000 UTC m=+51.768287655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls") pod "dns-default-msg6l" (UID: "a14c19c4-990c-4ec5-996c-bbaca8aafc93") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:30.169047 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.169016    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"]
Sep 17 23:11:30.170686 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.170670    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:11:30.175022 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.175000    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 23:11:30.175241 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.175226    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 23:11:30.175584 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.175568    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qsn4b\""
Sep 17 23:11:30.176134 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.176117    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 23:11:30.177739 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.177721    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 23:11:30.193277 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.193247    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"]
Sep 17 23:11:30.271038 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.271003    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-wvxp9"]
Sep 17 23:11:30.272687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.272672    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.276583 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.276562    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 23:11:30.276815 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.276800    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wl4k6\""
Sep 17 23:11:30.276895 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.276800    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 23:11:30.277429 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.277412    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 23:11:30.277523 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.277504    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 23:11:30.284648 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.284627    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 23:11:30.287861 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.287820    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-wvxp9"]
Sep 17 23:11:30.288707 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.288681    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q287s\" (UniqueName: \"kubernetes.io/projected/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-kube-api-access-q287s\") pod \"cluster-monitoring-operator-847b8bd8d7-qdg2c\" (UID: \"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:11:30.288821 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.288713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdg2c\" (UID: \"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:11:30.288821 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.288752    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-qdg2c\" (UID: \"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:11:30.390000 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.389964    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-qdg2c\" (UID: \"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:11:30.390177 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.390029    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2043c82f-26d1-4f3b-add6-385c35468db3-tmp\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.390177 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.390056    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2043c82f-26d1-4f3b-add6-385c35468db3-serving-cert\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.390177 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.390082    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2043c82f-26d1-4f3b-add6-385c35468db3-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.390177 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.390103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2043c82f-26d1-4f3b-add6-385c35468db3-snapshots\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.390355 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.390185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q287s\" (UniqueName: \"kubernetes.io/projected/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-kube-api-access-q287s\") pod \"cluster-monitoring-operator-847b8bd8d7-qdg2c\" (UID: \"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:11:30.390355 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.390226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2043c82f-26d1-4f3b-add6-385c35468db3-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.390355 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.390249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdg2c\" (UID: \"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:11:30.390355 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.390267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrpt8\" (UniqueName: \"kubernetes.io/projected/2043c82f-26d1-4f3b-add6-385c35468db3-kube-api-access-xrpt8\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.390355 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:30.390329    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:11:30.390572 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:30.390386    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls podName:2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:30.890373254 +0000 UTC m=+45.577445808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qdg2c" (UID: "2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:11:30.390736 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.390717    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-qdg2c\" (UID: \"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:11:30.411577 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.411541    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q287s\" (UniqueName: \"kubernetes.io/projected/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-kube-api-access-q287s\") pod \"cluster-monitoring-operator-847b8bd8d7-qdg2c\" (UID: \"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:11:30.490874 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.490762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2043c82f-26d1-4f3b-add6-385c35468db3-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.490874 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.490822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrpt8\" (UniqueName: \"kubernetes.io/projected/2043c82f-26d1-4f3b-add6-385c35468db3-kube-api-access-xrpt8\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.491049 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.490902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2043c82f-26d1-4f3b-add6-385c35468db3-tmp\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.491049 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.490921    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2043c82f-26d1-4f3b-add6-385c35468db3-serving-cert\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.491109 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.491075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2043c82f-26d1-4f3b-add6-385c35468db3-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.491139 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.491115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2043c82f-26d1-4f3b-add6-385c35468db3-snapshots\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.491347 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.491322    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2043c82f-26d1-4f3b-add6-385c35468db3-tmp\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.491562 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.491529    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2043c82f-26d1-4f3b-add6-385c35468db3-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.491696 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.491644    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2043c82f-26d1-4f3b-add6-385c35468db3-snapshots\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.491886 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.491868    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2043c82f-26d1-4f3b-add6-385c35468db3-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.493472 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.493452    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2043c82f-26d1-4f3b-add6-385c35468db3-serving-cert\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.500078 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.500056    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrpt8\" (UniqueName: \"kubernetes.io/projected/2043c82f-26d1-4f3b-add6-385c35468db3-kube-api-access-xrpt8\") pod \"insights-operator-6f9c644dd9-wvxp9\" (UID: \"2043c82f-26d1-4f3b-add6-385c35468db3\") " pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.581324 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.581263    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9"
Sep 17 23:11:30.856559 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.856530    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-wvxp9"]
Sep 17 23:11:30.858382 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:11:30.858343    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2043c82f_26d1_4f3b_add6_385c35468db3.slice/crio-ee7295ec58d7ec188b36fc6fc3b916cc6452cdc23ec9c1816e66e54b12871a0e WatchSource:0}: Error finding container ee7295ec58d7ec188b36fc6fc3b916cc6452cdc23ec9c1816e66e54b12871a0e: Status 404 returned error can't find the container with id ee7295ec58d7ec188b36fc6fc3b916cc6452cdc23ec9c1816e66e54b12871a0e
Sep 17 23:11:30.894445 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:30.894406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdg2c\" (UID: \"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:11:30.894598 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:30.894581    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:11:30.894658 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:30.894649    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls podName:2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:31.894631159 +0000 UTC m=+46.581703717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qdg2c" (UID: "2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:11:31.647911 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:31.647875    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9" event={"ID":"2043c82f-26d1-4f3b-add6-385c35468db3","Type":"ContainerStarted","Data":"ee7295ec58d7ec188b36fc6fc3b916cc6452cdc23ec9c1816e66e54b12871a0e"}
Sep 17 23:11:31.901764 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:31.901666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdg2c\" (UID: \"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:11:31.901959 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:31.901869    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:11:31.901959 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:31.901954    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls podName:2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:33.901933641 +0000 UTC m=+48.589006196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qdg2c" (UID: "2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:11:33.654753 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:33.654716    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9" event={"ID":"2043c82f-26d1-4f3b-add6-385c35468db3","Type":"ContainerStarted","Data":"df8a3392739c4fd507a5ab3e689b89e432facb5dc348bfe36e00ebbc31ed4d64"}
Sep 17 23:11:33.677983 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:33.677919    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9" podStartSLOduration=1.61086913 podStartE2EDuration="3.67789736s" totalImagesPullingTime="2.06702823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:33.677572927 +0000 UTC m=+48.364645504" watchObservedRunningTime="2026-09-17 23:11:33.67789736 +0000 UTC m=+48.364969932"
Sep 17 23:11:33.916352 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:33.916257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdg2c\" (UID: \"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:11:33.916533 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:33.916405    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:11:33.916533 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:33.916472    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls podName:2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:37.916452258 +0000 UTC m=+52.603524834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qdg2c" (UID: "2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:11:35.021450 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.021417    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f2sxs"]
Sep 17 23:11:35.024109 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.024089    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.027776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.027751    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fs4cd\""
Sep 17 23:11:35.027981 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.027966    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 23:11:35.028025 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.028001    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 23:11:35.041013 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.040989    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f2sxs"]
Sep 17 23:11:35.124356 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.124316    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/023d5e17-4668-4320-9949-9ec7f5a3feef-crio-socket\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.124525 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.124391    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.124525 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.124452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.124525 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.124515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/023d5e17-4668-4320-9949-9ec7f5a3feef-data-volume\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.124634 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.124533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2252z\" (UniqueName: \"kubernetes.io/projected/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-api-access-2252z\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.225900 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.225861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.226065 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.225922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.226065 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.225963    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/023d5e17-4668-4320-9949-9ec7f5a3feef-data-volume\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.226065 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:35.225924    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:35.226065 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.225989    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2252z\" (UniqueName: \"kubernetes.io/projected/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-api-access-2252z\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.226065 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:35.226006    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:35.226065 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.226048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/023d5e17-4668-4320-9949-9ec7f5a3feef-crio-socket\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.226065 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:35.226063    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm podName:023d5e17-4668-4320-9949-9ec7f5a3feef nodeName:}" failed. No retries permitted until 2026-09-17 23:11:35.72604475 +0000 UTC m=+50.413117305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2sxs" (UID: "023d5e17-4668-4320-9949-9ec7f5a3feef") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:35.226315 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:35.226111    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls podName:023d5e17-4668-4320-9949-9ec7f5a3feef nodeName:}" failed. No retries permitted until 2026-09-17 23:11:35.726085958 +0000 UTC m=+50.413158516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2sxs" (UID: "023d5e17-4668-4320-9949-9ec7f5a3feef") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:35.226315 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.226222    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/023d5e17-4668-4320-9949-9ec7f5a3feef-crio-socket\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.226315 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.226265    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/023d5e17-4668-4320-9949-9ec7f5a3feef-data-volume\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.237579 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.237550    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2252z\" (UniqueName: \"kubernetes.io/projected/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-api-access-2252z\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.729714 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.729668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.729714 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:35.729714    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:35.729970 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:35.729793    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:35.729970 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:35.729807    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:35.729970 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:35.729885    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm podName:023d5e17-4668-4320-9949-9ec7f5a3feef nodeName:}" failed. No retries permitted until 2026-09-17 23:11:36.729867797 +0000 UTC m=+51.416940353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2sxs" (UID: "023d5e17-4668-4320-9949-9ec7f5a3feef") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:35.729970 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:35.729901    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls podName:023d5e17-4668-4320-9949-9ec7f5a3feef nodeName:}" failed. No retries permitted until 2026-09-17 23:11:36.729894629 +0000 UTC m=+51.416967183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2sxs" (UID: "023d5e17-4668-4320-9949-9ec7f5a3feef") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:36.736918 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.736863    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:36.737383 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:36.737020    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:36.737383 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.737044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:36.737383 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:36.737084    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls podName:023d5e17-4668-4320-9949-9ec7f5a3feef nodeName:}" failed. No retries permitted until 2026-09-17 23:11:38.737069087 +0000 UTC m=+53.424141664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2sxs" (UID: "023d5e17-4668-4320-9949-9ec7f5a3feef") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:36.737383 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:36.737128    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:36.737383 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:36.737186    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm podName:023d5e17-4668-4320-9949-9ec7f5a3feef nodeName:}" failed. No retries permitted until 2026-09-17 23:11:38.737171497 +0000 UTC m=+53.424244055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2sxs" (UID: "023d5e17-4668-4320-9949-9ec7f5a3feef") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:36.738614 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.738591    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-pj5bs"]
Sep 17 23:11:36.740265 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.740251    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-pj5bs"
Sep 17 23:11:36.743587 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.743567    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 23:11:36.743701 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.743598    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 23:11:36.743701 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.743568    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 23:11:36.744643 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.744619    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tzhbs\""
Sep 17 23:11:36.744731 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.744696    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 23:11:36.754556 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.754525    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-pj5bs"]
Sep 17 23:11:36.838335 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.838293    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/81e5dc84-a5d4-41cf-b6d5-d794495f5cdd-signing-key\") pod \"service-ca-55f6fc7b86-pj5bs\" (UID: \"81e5dc84-a5d4-41cf-b6d5-d794495f5cdd\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pj5bs"
Sep 17 23:11:36.838535 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.838349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/81e5dc84-a5d4-41cf-b6d5-d794495f5cdd-signing-cabundle\") pod \"service-ca-55f6fc7b86-pj5bs\" (UID: \"81e5dc84-a5d4-41cf-b6d5-d794495f5cdd\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pj5bs"
Sep 17 23:11:36.838535 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.838374    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmmxc\" (UniqueName: \"kubernetes.io/projected/81e5dc84-a5d4-41cf-b6d5-d794495f5cdd-kube-api-access-zmmxc\") pod \"service-ca-55f6fc7b86-pj5bs\" (UID: \"81e5dc84-a5d4-41cf-b6d5-d794495f5cdd\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pj5bs"
Sep 17 23:11:36.939272 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.939233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/81e5dc84-a5d4-41cf-b6d5-d794495f5cdd-signing-cabundle\") pod \"service-ca-55f6fc7b86-pj5bs\" (UID: \"81e5dc84-a5d4-41cf-b6d5-d794495f5cdd\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pj5bs"
Sep 17 23:11:36.939272 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.939268    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmmxc\" (UniqueName: \"kubernetes.io/projected/81e5dc84-a5d4-41cf-b6d5-d794495f5cdd-kube-api-access-zmmxc\") pod \"service-ca-55f6fc7b86-pj5bs\" (UID: \"81e5dc84-a5d4-41cf-b6d5-d794495f5cdd\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pj5bs"
Sep 17 23:11:36.939515 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.939482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/81e5dc84-a5d4-41cf-b6d5-d794495f5cdd-signing-key\") pod \"service-ca-55f6fc7b86-pj5bs\" (UID: \"81e5dc84-a5d4-41cf-b6d5-d794495f5cdd\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pj5bs"
Sep 17 23:11:36.939916 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.939895    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/81e5dc84-a5d4-41cf-b6d5-d794495f5cdd-signing-cabundle\") pod \"service-ca-55f6fc7b86-pj5bs\" (UID: \"81e5dc84-a5d4-41cf-b6d5-d794495f5cdd\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pj5bs"
Sep 17 23:11:36.941969 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.941946    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/81e5dc84-a5d4-41cf-b6d5-d794495f5cdd-signing-key\") pod \"service-ca-55f6fc7b86-pj5bs\" (UID: \"81e5dc84-a5d4-41cf-b6d5-d794495f5cdd\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pj5bs"
Sep 17 23:11:36.953120 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.953099    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmmxc\" (UniqueName: \"kubernetes.io/projected/81e5dc84-a5d4-41cf-b6d5-d794495f5cdd-kube-api-access-zmmxc\") pod \"service-ca-55f6fc7b86-pj5bs\" (UID: \"81e5dc84-a5d4-41cf-b6d5-d794495f5cdd\") " pod="openshift-service-ca/service-ca-55f6fc7b86-pj5bs"
Sep 17 23:11:36.981723 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:36.981696    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-95nk9_d0fe30d8-d67c-4736-86fe-e33d93dafbb0/dns-node-resolver/0.log"
Sep 17 23:11:37.049522 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:37.049432    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-pj5bs"
Sep 17 23:11:37.141562 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:37.141522    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:37.141737 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:37.141653    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:37.141737 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:37.141712    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls podName:a14c19c4-990c-4ec5-996c-bbaca8aafc93 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:53.141699123 +0000 UTC m=+67.828771678 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls") pod "dns-default-msg6l" (UID: "a14c19c4-990c-4ec5-996c-bbaca8aafc93") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:37.323197 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:37.323109    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-pj5bs"]
Sep 17 23:11:37.326373 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:11:37.326345    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e5dc84_a5d4_41cf_b6d5_d794495f5cdd.slice/crio-9b450ff4f28c08f8888b3a2f1c27a2ac128abd19d02ac036c4bece7df9f93738 WatchSource:0}: Error finding container 9b450ff4f28c08f8888b3a2f1c27a2ac128abd19d02ac036c4bece7df9f93738: Status 404 returned error can't find the container with id 9b450ff4f28c08f8888b3a2f1c27a2ac128abd19d02ac036c4bece7df9f93738
Sep 17 23:11:37.664119 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:37.664083    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-pj5bs" event={"ID":"81e5dc84-a5d4-41cf-b6d5-d794495f5cdd","Type":"ContainerStarted","Data":"9b450ff4f28c08f8888b3a2f1c27a2ac128abd19d02ac036c4bece7df9f93738"}
Sep 17 23:11:37.783549 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:37.783522    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pxhs2_70ea03b1-9946-46d6-b68a-9c730d932c72/node-ca/0.log"
Sep 17 23:11:37.947435 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:37.947341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdg2c\" (UID: \"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:11:37.947596 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:37.947519    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:11:37.947640 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:37.947598    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls podName:2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:45.947577896 +0000 UTC m=+60.634650451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qdg2c" (UID: "2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:11:38.753745 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:38.753704    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:38.753955 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:38.753758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:38.753955 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:38.753874    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:38.753955 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:38.753888    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:38.754134 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:38.753957    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm podName:023d5e17-4668-4320-9949-9ec7f5a3feef nodeName:}" failed. No retries permitted until 2026-09-17 23:11:42.753933712 +0000 UTC m=+57.441006285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2sxs" (UID: "023d5e17-4668-4320-9949-9ec7f5a3feef") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:38.754134 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:38.753986    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls podName:023d5e17-4668-4320-9949-9ec7f5a3feef nodeName:}" failed. No retries permitted until 2026-09-17 23:11:42.753974895 +0000 UTC m=+57.441047454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2sxs" (UID: "023d5e17-4668-4320-9949-9ec7f5a3feef") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:38.787069 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:38.787034    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-st4l9_27fd2159-4f7f-477e-89f6-9fb97acf078a/serve-healthcheck-canary/0.log"
Sep 17 23:11:40.673790 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:40.673747    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-pj5bs" event={"ID":"81e5dc84-a5d4-41cf-b6d5-d794495f5cdd","Type":"ContainerStarted","Data":"2e1fb584c927e6f51901c44c27e2533ddb2895ece18da8f7037b57f7fd67ac1e"}
Sep 17 23:11:42.788629 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:42.788565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:42.788629 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:42.788615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:42.789082 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:42.788714    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:42.789082 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:42.788732    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:42.789082 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:42.788794    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls podName:023d5e17-4668-4320-9949-9ec7f5a3feef nodeName:}" failed. No retries permitted until 2026-09-17 23:11:50.788777907 +0000 UTC m=+65.475850465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2sxs" (UID: "023d5e17-4668-4320-9949-9ec7f5a3feef") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:42.789082 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:42.788811    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm podName:023d5e17-4668-4320-9949-9ec7f5a3feef nodeName:}" failed. No retries permitted until 2026-09-17 23:11:50.788802559 +0000 UTC m=+65.475875113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2sxs" (UID: "023d5e17-4668-4320-9949-9ec7f5a3feef") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:46.018750 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:46.018707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdg2c\" (UID: \"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:11:46.019154 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:46.018905    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:11:46.019154 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:46.018989    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls podName:2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:02.018972288 +0000 UTC m=+76.706044856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-qdg2c" (UID: "2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:11:47.612047 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:47.612017    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-88d87"
Sep 17 23:11:47.663043 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:47.662979    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-pj5bs" podStartSLOduration=9.166773385 podStartE2EDuration="11.662958945s" totalImagesPullingTime="2.49618556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:40.702828258 +0000 UTC m=+55.389900836" watchObservedRunningTime="2026-09-17 23:11:47.662958945 +0000 UTC m=+62.350031523"
Sep 17 23:11:50.859701 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:50.859659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:50.859701 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:50.859708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:50.860289 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:50.859887    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:50.860289 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:11:50.859961    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls podName:023d5e17-4668-4320-9949-9ec7f5a3feef nodeName:}" failed. No retries permitted until 2026-09-17 23:12:06.859943542 +0000 UTC m=+81.547016120 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2sxs" (UID: "023d5e17-4668-4320-9949-9ec7f5a3feef") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:50.860408 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:50.860373    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/023d5e17-4668-4320-9949-9ec7f5a3feef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:11:53.074203 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.074138    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:53.074203 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.074213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs\") pod \"network-metrics-daemon-nmn9p\" (UID: \"80efd7c7-aa08-47ee-a917-251ef8624493\") " pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:53.078080 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.078058    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:11:53.079107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.079085    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:11:53.086887 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.086861    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d48fd517-45d0-445c-836e-7045570e3385-original-pull-secret\") pod \"global-pull-secret-syncer-s7lg2\" (UID: \"d48fd517-45d0-445c-836e-7045570e3385\") " pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:53.086988 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.086934    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80efd7c7-aa08-47ee-a917-251ef8624493-metrics-certs\") pod \"network-metrics-daemon-nmn9p\" (UID: \"80efd7c7-aa08-47ee-a917-251ef8624493\") " pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:53.175286 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.175248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tjl\" (UniqueName: \"kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl\") pod \"network-check-target-5xnb7\" (UID: \"9e2add4f-c624-4d66-a7c5-8136a8ff8943\") " pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:53.175286 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.175289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:53.177827 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.177803    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a14c19c4-990c-4ec5-996c-bbaca8aafc93-metrics-tls\") pod \"dns-default-msg6l\" (UID: \"a14c19c4-990c-4ec5-996c-bbaca8aafc93\") " pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:53.178769 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.178748    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:11:53.188910 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.188887    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:11:53.198731 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.198701    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9tjl\" (UniqueName: \"kubernetes.io/projected/9e2add4f-c624-4d66-a7c5-8136a8ff8943-kube-api-access-m9tjl\") pod \"network-check-target-5xnb7\" (UID: \"9e2add4f-c624-4d66-a7c5-8136a8ff8943\") " pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:53.204510 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.204489    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k7845\""
Sep 17 23:11:53.210029 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.210013    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jknmn\""
Sep 17 23:11:53.211056 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.211040    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmn9p"
Sep 17 23:11:53.212226 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.212209    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s7lg2"
Sep 17 23:11:53.218423 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.218408    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:53.384183 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.384155    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-42llv\""
Sep 17 23:11:53.389097 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.389077    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:53.517150 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.517117    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5xnb7"]
Sep 17 23:11:53.524674 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.521811    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nmn9p"]
Sep 17 23:11:53.524674 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.523314    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s7lg2"]
Sep 17 23:11:53.663429 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.663406    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-msg6l"]
Sep 17 23:11:53.664783 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:11:53.664756    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14c19c4_990c_4ec5_996c_bbaca8aafc93.slice/crio-0e397d32055da66953da5d3f25d9fbe25e5b06529f8c65b6ab34c0bfbfd31dbf WatchSource:0}: Error finding container 0e397d32055da66953da5d3f25d9fbe25e5b06529f8c65b6ab34c0bfbfd31dbf: Status 404 returned error can't find the container with id 0e397d32055da66953da5d3f25d9fbe25e5b06529f8c65b6ab34c0bfbfd31dbf
Sep 17 23:11:53.708788 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.708752    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-msg6l" event={"ID":"a14c19c4-990c-4ec5-996c-bbaca8aafc93","Type":"ContainerStarted","Data":"0e397d32055da66953da5d3f25d9fbe25e5b06529f8c65b6ab34c0bfbfd31dbf"}
Sep 17 23:11:53.709899 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.709864    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5xnb7" event={"ID":"9e2add4f-c624-4d66-a7c5-8136a8ff8943","Type":"ContainerStarted","Data":"30b4460404f4c4362f7c85e51f7a9186f00f8c9a813f44debd5a7caab90f4ae2"}
Sep 17 23:11:53.710908 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.710883    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s7lg2" event={"ID":"d48fd517-45d0-445c-836e-7045570e3385","Type":"ContainerStarted","Data":"d0970112a45ec549f82cf58615415c446c131fd8af371eb3621db1abea1be617"}
Sep 17 23:11:53.711859 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:53.711823    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmn9p" event={"ID":"80efd7c7-aa08-47ee-a917-251ef8624493","Type":"ContainerStarted","Data":"f938bdafbbb2b060824ac758def223ef80915441bc9e28c7e0fc1586d93ae809"}
Sep 17 23:11:55.722294 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:55.722255    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmn9p" event={"ID":"80efd7c7-aa08-47ee-a917-251ef8624493","Type":"ContainerStarted","Data":"83e31ec67b54afd7db4a09fe542a8bee97a0079a9c7037d8ce77f7584febc363"}
Sep 17 23:11:55.722771 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:55.722325    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmn9p" event={"ID":"80efd7c7-aa08-47ee-a917-251ef8624493","Type":"ContainerStarted","Data":"4f8533647ca73cc6b034ca3dc774e21a6478035f1c315d554ca933b723fcffd8"}
Sep 17 23:11:55.744240 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:55.743187    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nmn9p" podStartSLOduration=67.490315808 podStartE2EDuration="1m8.743159925s" totalImagesPullingTime="1.252844117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:55.742747456 +0000 UTC m=+70.429820034" watchObservedRunningTime="2026-09-17 23:11:55.743159925 +0000 UTC m=+70.430232503"
Sep 17 23:11:58.733543 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:58.733498    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-msg6l" event={"ID":"a14c19c4-990c-4ec5-996c-bbaca8aafc93","Type":"ContainerStarted","Data":"634dbee2fb147c191c965b2c7624e67419cfc7c71c632f7d7e2fd99ba9ec5272"}
Sep 17 23:11:58.734065 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:58.733680    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-msg6l"
Sep 17 23:11:58.734065 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:58.733711    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-msg6l" event={"ID":"a14c19c4-990c-4ec5-996c-bbaca8aafc93","Type":"ContainerStarted","Data":"68612560ca117345d1e90bd6b2cc8655b8983fbd02396a9c24af9a0300d808af"}
Sep 17 23:11:58.735005 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:58.734985    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5xnb7" event={"ID":"9e2add4f-c624-4d66-a7c5-8136a8ff8943","Type":"ContainerStarted","Data":"4bd97ad1f559d656439ebfd516b4ccd2f8155f4c25c617f7335939a6af45e5f6"}
Sep 17 23:11:58.735072 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:58.735052    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:11:58.736434 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:58.736407    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s7lg2" event={"ID":"d48fd517-45d0-445c-836e-7045570e3385","Type":"ContainerStarted","Data":"230a3cd4de59d4b1c3a2b5e473418ddc8dfa79d7c93d5423d263cfb4922ebffb"}
Sep 17 23:11:58.770529 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:58.770476    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-msg6l" podStartSLOduration=35.629282077 podStartE2EDuration="37.770459342s" totalImagesPullingTime="2.141177265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:58.769769336 +0000 UTC m=+73.456841914" watchObservedRunningTime="2026-09-17 23:11:58.770459342 +0000 UTC m=+73.457531948"
Sep 17 23:11:58.794906 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:58.794854    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-s7lg2" podStartSLOduration=67.072260824 podStartE2EDuration="1m11.794812951s" totalImagesPullingTime="4.722552127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:58.794389077 +0000 UTC m=+73.481461655" watchObservedRunningTime="2026-09-17 23:11:58.794812951 +0000 UTC m=+73.481885529"
Sep 17 23:11:58.820877 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:11:58.820801    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5xnb7" podStartSLOduration=67.101455882 podStartE2EDuration="1m11.820783161s" totalImagesPullingTime="4.719327279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:58.82023028 +0000 UTC m=+73.507302858" watchObservedRunningTime="2026-09-17 23:11:58.820783161 +0000 UTC m=+73.507855785"
Sep 17 23:12:01.697270 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.697235    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h"]
Sep 17 23:12:01.699137 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.699121    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7jbxb"]
Sep 17 23:12:01.699260 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.699241    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h"
Sep 17 23:12:01.701066 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.701043    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jbxb"
Sep 17 23:12:01.701949 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.701933    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 23:12:01.702036 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.701933    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-dz8d7\""
Sep 17 23:12:01.702988 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.702959    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 23:12:01.703092 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.702995    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 23:12:01.703092 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.703015    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:12:01.703758 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.703745    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 23:12:01.704195 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.704175    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 23:12:01.705423 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.705405    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-h9rc4\""
Sep 17 23:12:01.724490 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.724468    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h"]
Sep 17 23:12:01.725245 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.725223    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7jbxb"]
Sep 17 23:12:01.745078 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.745050    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsmzv\" (UniqueName: \"kubernetes.io/projected/54abc83b-5baf-48fa-b91f-72d90b85aca6-kube-api-access-zsmzv\") pod \"kube-storage-version-migrator-operator-55889f7b47-6786h\" (UID: \"54abc83b-5baf-48fa-b91f-72d90b85aca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h"
Sep 17 23:12:01.745218 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.745083    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4bdccb4e-063e-4cda-8ed8-b36b36bf2643-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7jbxb\" (UID: \"4bdccb4e-063e-4cda-8ed8-b36b36bf2643\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jbxb"
Sep 17 23:12:01.745218 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.745151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/54abc83b-5baf-48fa-b91f-72d90b85aca6-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-6786h\" (UID: \"54abc83b-5baf-48fa-b91f-72d90b85aca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h"
Sep 17 23:12:01.745293 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.745218    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/54abc83b-5baf-48fa-b91f-72d90b85aca6-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-6786h\" (UID: \"54abc83b-5baf-48fa-b91f-72d90b85aca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h"
Sep 17 23:12:01.745293 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.745245    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4bdccb4e-063e-4cda-8ed8-b36b36bf2643-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7jbxb\" (UID: \"4bdccb4e-063e-4cda-8ed8-b36b36bf2643\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jbxb"
Sep 17 23:12:01.798306 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.798273    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hcp48"]
Sep 17 23:12:01.800167 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.800145    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-bzzqr"]
Sep 17 23:12:01.800304 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.800263    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hcp48"
Sep 17 23:12:01.801999 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.801977    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-bzzqr"
Sep 17 23:12:01.804097 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.804076    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"]
Sep 17 23:12:01.805795 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.805767    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.806641 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.806626    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-687jg\""
Sep 17 23:12:01.808117 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.808098    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 23:12:01.808205 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.808099    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-475cn\""
Sep 17 23:12:01.808773 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.808759    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 23:12:01.809186 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.809168    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:12:01.809329 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.809235    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 23:12:01.809457 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.809238    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 23:12:01.809457 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.809248    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 23:12:01.809457 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.809259    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:12:01.809680 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.809272    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 23:12:01.809734 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.809705    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 23:12:01.809784 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.809771    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 23:12:01.809861 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.809790    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 23:12:01.809861 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.809806    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 23:12:01.830236 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.830210    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hcp48"]
Sep 17 23:12:01.831075 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.831057    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"]
Sep 17 23:12:01.841182 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.841155    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-bzzqr"]
Sep 17 23:12:01.846255 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.846230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/54abc83b-5baf-48fa-b91f-72d90b85aca6-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-6786h\" (UID: \"54abc83b-5baf-48fa-b91f-72d90b85aca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h"
Sep 17 23:12:01.846368 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.846279    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqkk9\" (UniqueName: \"kubernetes.io/projected/5b2108d0-67c1-40c4-9fcf-2948603ae05e-kube-api-access-sqkk9\") pod \"cluster-samples-operator-685d7d68b5-bzzqr\" (UID: \"5b2108d0-67c1-40c4-9fcf-2948603ae05e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-bzzqr"
Sep 17 23:12:01.846368 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.846340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmz5h\" (UniqueName: \"kubernetes.io/projected/712b2a11-e159-4115-836d-9270325d32e0-kube-api-access-tmz5h\") pod \"volume-data-source-validator-5c4db8bdf9-hcp48\" (UID: \"712b2a11-e159-4115-836d-9270325d32e0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hcp48"
Sep 17 23:12:01.846453 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.846386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b2108d0-67c1-40c4-9fcf-2948603ae05e-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-bzzqr\" (UID: \"5b2108d0-67c1-40c4-9fcf-2948603ae05e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-bzzqr"
Sep 17 23:12:01.846453 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.846411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/737e63db-9417-409c-9dec-691fd5cb79e4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.846523 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.846458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4bdccb4e-063e-4cda-8ed8-b36b36bf2643-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7jbxb\" (UID: \"4bdccb4e-063e-4cda-8ed8-b36b36bf2643\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jbxb"
Sep 17 23:12:01.846523 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.846499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/54abc83b-5baf-48fa-b91f-72d90b85aca6-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-6786h\" (UID: \"54abc83b-5baf-48fa-b91f-72d90b85aca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h"
Sep 17 23:12:01.846612 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.846537    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsmzv\" (UniqueName: \"kubernetes.io/projected/54abc83b-5baf-48fa-b91f-72d90b85aca6-kube-api-access-zsmzv\") pod \"kube-storage-version-migrator-operator-55889f7b47-6786h\" (UID: \"54abc83b-5baf-48fa-b91f-72d90b85aca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h"
Sep 17 23:12:01.846612 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.846562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/737e63db-9417-409c-9dec-691fd5cb79e4-ca\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.846612 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.846578    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/737e63db-9417-409c-9dec-691fd5cb79e4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.846754 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.846695    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dhtb\" (UniqueName: \"kubernetes.io/projected/737e63db-9417-409c-9dec-691fd5cb79e4-kube-api-access-8dhtb\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.846754 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.846743    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4bdccb4e-063e-4cda-8ed8-b36b36bf2643-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7jbxb\" (UID: \"4bdccb4e-063e-4cda-8ed8-b36b36bf2643\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jbxb"
Sep 17 23:12:01.846879 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.846793    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/737e63db-9417-409c-9dec-691fd5cb79e4-hub\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.846879 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.846827    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/737e63db-9417-409c-9dec-691fd5cb79e4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.847245 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.847186    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/54abc83b-5baf-48fa-b91f-72d90b85aca6-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-6786h\" (UID: \"54abc83b-5baf-48fa-b91f-72d90b85aca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h"
Sep 17 23:12:01.847447 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.847425    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4bdccb4e-063e-4cda-8ed8-b36b36bf2643-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-7jbxb\" (UID: \"4bdccb4e-063e-4cda-8ed8-b36b36bf2643\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jbxb"
Sep 17 23:12:01.848977 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.848958    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/54abc83b-5baf-48fa-b91f-72d90b85aca6-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-6786h\" (UID: \"54abc83b-5baf-48fa-b91f-72d90b85aca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h"
Sep 17 23:12:01.849058 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.849005    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4bdccb4e-063e-4cda-8ed8-b36b36bf2643-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-7jbxb\" (UID: \"4bdccb4e-063e-4cda-8ed8-b36b36bf2643\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jbxb"
Sep 17 23:12:01.948248 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.948142    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqkk9\" (UniqueName: \"kubernetes.io/projected/5b2108d0-67c1-40c4-9fcf-2948603ae05e-kube-api-access-sqkk9\") pod \"cluster-samples-operator-685d7d68b5-bzzqr\" (UID: \"5b2108d0-67c1-40c4-9fcf-2948603ae05e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-bzzqr"
Sep 17 23:12:01.948248 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.948218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmz5h\" (UniqueName: \"kubernetes.io/projected/712b2a11-e159-4115-836d-9270325d32e0-kube-api-access-tmz5h\") pod \"volume-data-source-validator-5c4db8bdf9-hcp48\" (UID: \"712b2a11-e159-4115-836d-9270325d32e0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hcp48"
Sep 17 23:12:01.948502 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.948256    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b2108d0-67c1-40c4-9fcf-2948603ae05e-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-bzzqr\" (UID: \"5b2108d0-67c1-40c4-9fcf-2948603ae05e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-bzzqr"
Sep 17 23:12:01.948502 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.948280    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/737e63db-9417-409c-9dec-691fd5cb79e4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.948502 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.948334    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/737e63db-9417-409c-9dec-691fd5cb79e4-ca\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.948502 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.948356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/737e63db-9417-409c-9dec-691fd5cb79e4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.948502 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.948383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dhtb\" (UniqueName: \"kubernetes.io/projected/737e63db-9417-409c-9dec-691fd5cb79e4-kube-api-access-8dhtb\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.948502 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.948417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/737e63db-9417-409c-9dec-691fd5cb79e4-hub\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.948922 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.948546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/737e63db-9417-409c-9dec-691fd5cb79e4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.949266 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.949241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/737e63db-9417-409c-9dec-691fd5cb79e4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.951285 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.951258    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/737e63db-9417-409c-9dec-691fd5cb79e4-hub\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.951487 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.951469    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/737e63db-9417-409c-9dec-691fd5cb79e4-ca\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.951561 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.951544    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/737e63db-9417-409c-9dec-691fd5cb79e4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:01.951690 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.951668    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b2108d0-67c1-40c4-9fcf-2948603ae05e-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-bzzqr\" (UID: \"5b2108d0-67c1-40c4-9fcf-2948603ae05e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-bzzqr"
Sep 17 23:12:01.951762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:01.951747    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/737e63db-9417-409c-9dec-691fd5cb79e4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:02.003880 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.003826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqkk9\" (UniqueName: \"kubernetes.io/projected/5b2108d0-67c1-40c4-9fcf-2948603ae05e-kube-api-access-sqkk9\") pod \"cluster-samples-operator-685d7d68b5-bzzqr\" (UID: \"5b2108d0-67c1-40c4-9fcf-2948603ae05e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-bzzqr"
Sep 17 23:12:02.007104 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.007083    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsmzv\" (UniqueName: \"kubernetes.io/projected/54abc83b-5baf-48fa-b91f-72d90b85aca6-kube-api-access-zsmzv\") pod \"kube-storage-version-migrator-operator-55889f7b47-6786h\" (UID: \"54abc83b-5baf-48fa-b91f-72d90b85aca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h"
Sep 17 23:12:02.010010 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.009987    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h"
Sep 17 23:12:02.013889 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.013870    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5999d9d4c4-8hg92"]
Sep 17 23:12:02.014917 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.014903    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jbxb"
Sep 17 23:12:02.015605 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.015592    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.049965 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.049906    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdg2c\" (UID: \"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:12:02.050163 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.049985    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9abe976-07fe-4a06-be0e-530da46ae4eb-installation-pull-secrets\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.050163 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.050022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9abe976-07fe-4a06-be0e-530da46ae4eb-image-registry-private-configuration\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.050163 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.050052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9abe976-07fe-4a06-be0e-530da46ae4eb-registry-certificates\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.050163 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.050140    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9abe976-07fe-4a06-be0e-530da46ae4eb-ca-trust-extracted\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.050340 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.050187    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw8mh\" (UniqueName: \"kubernetes.io/projected/c9abe976-07fe-4a06-be0e-530da46ae4eb-kube-api-access-kw8mh\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.050340 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.050231    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9abe976-07fe-4a06-be0e-530da46ae4eb-registry-tls\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.050340 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.050255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9abe976-07fe-4a06-be0e-530da46ae4eb-trusted-ca\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.050340 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.050315    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9abe976-07fe-4a06-be0e-530da46ae4eb-bound-sa-token\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.052746 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.052706    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-qdg2c\" (UID: \"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:12:02.102094 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.102060    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 23:12:02.102640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.102614    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5987r\""
Sep 17 23:12:02.102767 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.102684    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 23:12:02.103192 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.103173    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 23:12:02.105032 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.105008    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmz5h\" (UniqueName: \"kubernetes.io/projected/712b2a11-e159-4115-836d-9270325d32e0-kube-api-access-tmz5h\") pod \"volume-data-source-validator-5c4db8bdf9-hcp48\" (UID: \"712b2a11-e159-4115-836d-9270325d32e0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hcp48"
Sep 17 23:12:02.105296 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.105273    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dhtb\" (UniqueName: \"kubernetes.io/projected/737e63db-9417-409c-9dec-691fd5cb79e4-kube-api-access-8dhtb\") pod \"cluster-proxy-proxy-agent-67b4df54b9-6sscs\" (UID: \"737e63db-9417-409c-9dec-691fd5cb79e4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:02.111088 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.111069    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hcp48"
Sep 17 23:12:02.116678 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.116661    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-bzzqr"
Sep 17 23:12:02.126669 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.126648    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 23:12:02.132135 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.132110    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"
Sep 17 23:12:02.151983 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.151935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9abe976-07fe-4a06-be0e-530da46ae4eb-installation-pull-secrets\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.152148 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.152011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9abe976-07fe-4a06-be0e-530da46ae4eb-image-registry-private-configuration\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.152148 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.152047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9abe976-07fe-4a06-be0e-530da46ae4eb-registry-certificates\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.152148 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.152075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9abe976-07fe-4a06-be0e-530da46ae4eb-ca-trust-extracted\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.152148 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.152105    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw8mh\" (UniqueName: \"kubernetes.io/projected/c9abe976-07fe-4a06-be0e-530da46ae4eb-kube-api-access-kw8mh\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.152377 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.152150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9abe976-07fe-4a06-be0e-530da46ae4eb-registry-tls\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.152377 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.152172    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9abe976-07fe-4a06-be0e-530da46ae4eb-trusted-ca\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.152377 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.152202    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9abe976-07fe-4a06-be0e-530da46ae4eb-bound-sa-token\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.153096 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.152735    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9abe976-07fe-4a06-be0e-530da46ae4eb-ca-trust-extracted\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.153525 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.153494    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9abe976-07fe-4a06-be0e-530da46ae4eb-registry-certificates\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.154474 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.154369    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9abe976-07fe-4a06-be0e-530da46ae4eb-trusted-ca\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.155930 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.155858    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9abe976-07fe-4a06-be0e-530da46ae4eb-registry-tls\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.156548 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.156480    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9abe976-07fe-4a06-be0e-530da46ae4eb-image-registry-private-configuration\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.156852 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.156815    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9abe976-07fe-4a06-be0e-530da46ae4eb-installation-pull-secrets\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.209276 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.209149    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9abe976-07fe-4a06-be0e-530da46ae4eb-bound-sa-token\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.213329 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.213102    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5999d9d4c4-8hg92"]
Sep 17 23:12:02.219319 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.219254    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw8mh\" (UniqueName: \"kubernetes.io/projected/c9abe976-07fe-4a06-be0e-530da46ae4eb-kube-api-access-kw8mh\") pod \"image-registry-5999d9d4c4-8hg92\" (UID: \"c9abe976-07fe-4a06-be0e-530da46ae4eb\") " pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.285726 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.285694    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qsn4b\""
Sep 17 23:12:02.290792 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.290770    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"
Sep 17 23:12:02.325520 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.325488    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.332559 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.332537    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h"]
Sep 17 23:12:02.333709 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.333689    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-7jbxb"]
Sep 17 23:12:02.334350 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:12:02.334315    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdccb4e_063e_4cda_8ed8_b36b36bf2643.slice/crio-4d43bbbf23919d61df59ac1ce59c60c5ccf419b90e76e6df0d1ca91700990f90 WatchSource:0}: Error finding container 4d43bbbf23919d61df59ac1ce59c60c5ccf419b90e76e6df0d1ca91700990f90: Status 404 returned error can't find the container with id 4d43bbbf23919d61df59ac1ce59c60c5ccf419b90e76e6df0d1ca91700990f90
Sep 17 23:12:02.335323 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:12:02.335282    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54abc83b_5baf_48fa_b91f_72d90b85aca6.slice/crio-6f4fad69dac5075a34e3828b052791f344b1620750ea803f1bb30c050fe12038 WatchSource:0}: Error finding container 6f4fad69dac5075a34e3828b052791f344b1620750ea803f1bb30c050fe12038: Status 404 returned error can't find the container with id 6f4fad69dac5075a34e3828b052791f344b1620750ea803f1bb30c050fe12038
Sep 17 23:12:02.427016 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.426979    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hcp48"]
Sep 17 23:12:02.428023 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.428000    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-bzzqr"]
Sep 17 23:12:02.429407 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:12:02.429381    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712b2a11_e159_4115_836d_9270325d32e0.slice/crio-3e86eb48c8029eca2e55c92e64934c8b6d58ba499313bd6382ec29db5e0040bb WatchSource:0}: Error finding container 3e86eb48c8029eca2e55c92e64934c8b6d58ba499313bd6382ec29db5e0040bb: Status 404 returned error can't find the container with id 3e86eb48c8029eca2e55c92e64934c8b6d58ba499313bd6382ec29db5e0040bb
Sep 17 23:12:02.458967 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.458942    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs"]
Sep 17 23:12:02.461295 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:12:02.461266    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod737e63db_9417_409c_9dec_691fd5cb79e4.slice/crio-9621da61713bc36f831e8e7edb3aeb7facc99b2e235b1d0d99f2366d0d96d3cb WatchSource:0}: Error finding container 9621da61713bc36f831e8e7edb3aeb7facc99b2e235b1d0d99f2366d0d96d3cb: Status 404 returned error can't find the container with id 9621da61713bc36f831e8e7edb3aeb7facc99b2e235b1d0d99f2366d0d96d3cb
Sep 17 23:12:02.566582 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.566544    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c"]
Sep 17 23:12:02.569614 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:12:02.569583    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2329ed7e_d5f9_46dc_a77c_14ef9dae6fc0.slice/crio-9a817c4b94a82dc68d0178a84c0e504af4944873ca365c5cc5f038c01082c4f5 WatchSource:0}: Error finding container 9a817c4b94a82dc68d0178a84c0e504af4944873ca365c5cc5f038c01082c4f5: Status 404 returned error can't find the container with id 9a817c4b94a82dc68d0178a84c0e504af4944873ca365c5cc5f038c01082c4f5
Sep 17 23:12:02.644877 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.644823    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5999d9d4c4-8hg92"]
Sep 17 23:12:02.647849 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:12:02.647801    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9abe976_07fe_4a06_be0e_530da46ae4eb.slice/crio-257264b517e2e4b8617d9a5fcd9157adbf2bc8ed420b8a04d2ff0988b4172567 WatchSource:0}: Error finding container 257264b517e2e4b8617d9a5fcd9157adbf2bc8ed420b8a04d2ff0988b4172567: Status 404 returned error can't find the container with id 257264b517e2e4b8617d9a5fcd9157adbf2bc8ed420b8a04d2ff0988b4172567
Sep 17 23:12:02.751486 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.751395    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs" event={"ID":"737e63db-9417-409c-9dec-691fd5cb79e4","Type":"ContainerStarted","Data":"9621da61713bc36f831e8e7edb3aeb7facc99b2e235b1d0d99f2366d0d96d3cb"}
Sep 17 23:12:02.753895 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.753859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92" event={"ID":"c9abe976-07fe-4a06-be0e-530da46ae4eb","Type":"ContainerStarted","Data":"f22cd8ea3a4248f463c1a31b36b76995b2bb61b97180043ce1067ff90193828b"}
Sep 17 23:12:02.754069 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.753918    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92" event={"ID":"c9abe976-07fe-4a06-be0e-530da46ae4eb","Type":"ContainerStarted","Data":"257264b517e2e4b8617d9a5fcd9157adbf2bc8ed420b8a04d2ff0988b4172567"}
Sep 17 23:12:02.754069 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.753986    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:02.755727 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.755698    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hcp48" event={"ID":"712b2a11-e159-4115-836d-9270325d32e0","Type":"ContainerStarted","Data":"3e86eb48c8029eca2e55c92e64934c8b6d58ba499313bd6382ec29db5e0040bb"}
Sep 17 23:12:02.757429 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.757373    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jbxb" event={"ID":"4bdccb4e-063e-4cda-8ed8-b36b36bf2643","Type":"ContainerStarted","Data":"4d43bbbf23919d61df59ac1ce59c60c5ccf419b90e76e6df0d1ca91700990f90"}
Sep 17 23:12:02.759045 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.759018    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h" event={"ID":"54abc83b-5baf-48fa-b91f-72d90b85aca6","Type":"ContainerStarted","Data":"6f4fad69dac5075a34e3828b052791f344b1620750ea803f1bb30c050fe12038"}
Sep 17 23:12:02.760674 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.760643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c" event={"ID":"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0","Type":"ContainerStarted","Data":"9a817c4b94a82dc68d0178a84c0e504af4944873ca365c5cc5f038c01082c4f5"}
Sep 17 23:12:02.761991 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.761967    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-bzzqr" event={"ID":"5b2108d0-67c1-40c4-9fcf-2948603ae05e","Type":"ContainerStarted","Data":"37691efd368eb4b047dc0b1488dfe563726cb83036944735cbde917a73a39691"}
Sep 17 23:12:02.788795 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:02.788736    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92" podStartSLOduration=1.788717519 podStartE2EDuration="1.788717519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:02.785622098 +0000 UTC m=+77.472694676" watchObservedRunningTime="2026-09-17 23:12:02.788717519 +0000 UTC m=+77.475790129"
Sep 17 23:12:06.898980 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:06.898947    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:12:06.901678 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:06.901646    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/023d5e17-4668-4320-9949-9ec7f5a3feef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2sxs\" (UID: \"023d5e17-4668-4320-9949-9ec7f5a3feef\") " pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:12:07.157216 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:07.157184    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fs4cd\""
Sep 17 23:12:07.164321 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:07.164290    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f2sxs"
Sep 17 23:12:08.336544 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:12:08.336503    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023d5e17_4668_4320_9949_9ec7f5a3feef.slice/crio-968da33566b54f07cc0a2947d6d7123626d6f8b813a72c89d0c5cbc07577cf44 WatchSource:0}: Error finding container 968da33566b54f07cc0a2947d6d7123626d6f8b813a72c89d0c5cbc07577cf44: Status 404 returned error can't find the container with id 968da33566b54f07cc0a2947d6d7123626d6f8b813a72c89d0c5cbc07577cf44
Sep 17 23:12:08.337988 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:08.337958    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f2sxs"]
Sep 17 23:12:08.743626 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:08.743590    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-msg6l"
Sep 17 23:12:08.799033 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:08.798447    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs" event={"ID":"737e63db-9417-409c-9dec-691fd5cb79e4","Type":"ContainerStarted","Data":"ba8d9565aa15c694d3335480878804c3fbf6c0c979bedc4e08feb4bd9f725afa"}
Sep 17 23:12:08.799033 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:08.798503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs" event={"ID":"737e63db-9417-409c-9dec-691fd5cb79e4","Type":"ContainerStarted","Data":"9d21928acd7a76d380b8a693eaf3ec5ac5b082039dbbf02d79d110e8f8ba39dc"}
Sep 17 23:12:08.799033 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:08.798520    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs" event={"ID":"737e63db-9417-409c-9dec-691fd5cb79e4","Type":"ContainerStarted","Data":"42f7f7e63592d81a86521d9ccbb3823743f58ad97399ea597ce9dff17d1159bd"}
Sep 17 23:12:08.802404 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:08.802368    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2sxs" event={"ID":"023d5e17-4668-4320-9949-9ec7f5a3feef","Type":"ContainerStarted","Data":"10f7085a528a9fdeb1314f77588a009996bac29d4d55109c71d41953dc7ad4aa"}
Sep 17 23:12:08.802566 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:08.802433    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2sxs" event={"ID":"023d5e17-4668-4320-9949-9ec7f5a3feef","Type":"ContainerStarted","Data":"968da33566b54f07cc0a2947d6d7123626d6f8b813a72c89d0c5cbc07577cf44"}
Sep 17 23:12:08.804303 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:08.804267    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hcp48" event={"ID":"712b2a11-e159-4115-836d-9270325d32e0","Type":"ContainerStarted","Data":"27126bc882da4a300370c9f046189b4df362cd3753cca87e49e3dc824dbe7ff1"}
Sep 17 23:12:08.807574 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:08.807545    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jbxb" event={"ID":"4bdccb4e-063e-4cda-8ed8-b36b36bf2643","Type":"ContainerStarted","Data":"79b850b1575ddd03d4c3494847b7fe7ec4121c57f9ac309305529bdb13b79f9f"}
Sep 17 23:12:08.809379 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:08.809357    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h" event={"ID":"54abc83b-5baf-48fa-b91f-72d90b85aca6","Type":"ContainerStarted","Data":"77d27ddb2300c1625807ef257d799071789aaf7f26aeb6cbc380c0bfd404b8cd"}
Sep 17 23:12:08.811774 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:08.811749    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c" event={"ID":"2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0","Type":"ContainerStarted","Data":"3914100c1e2225ee08f12cb96281daecdf009c304015b5e2bb69bed90f2007a2"}
Sep 17 23:12:08.814008 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:08.813923    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-bzzqr" event={"ID":"5b2108d0-67c1-40c4-9fcf-2948603ae05e","Type":"ContainerStarted","Data":"3fd5ce8db4cee6512a27e6c3ac5a6d3c22b77d456428062e4d298834b6e08e3b"}
Sep 17 23:12:08.814008 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:08.813975    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-bzzqr" event={"ID":"5b2108d0-67c1-40c4-9fcf-2948603ae05e","Type":"ContainerStarted","Data":"ff4dcdc073ec36fdad9eb7fcbaec550eb777ede734363bfa0ec57d24ac4722a1"}
Sep 17 23:12:08.853765 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:08.853526    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hcp48" podStartSLOduration=2.311611798 podStartE2EDuration="7.853501806s" totalImagesPullingTime="5.541890008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:08.850390994 +0000 UTC m=+83.537463585" watchObservedRunningTime="2026-09-17 23:12:08.853501806 +0000 UTC m=+83.540574381"
Sep 17 23:12:08.918973 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:08.918922    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67b4df54b9-6sscs" podStartSLOduration=2.377830665 podStartE2EDuration="7.918901634s" totalImagesPullingTime="5.541070969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:08.918668539 +0000 UTC m=+83.605741117" watchObservedRunningTime="2026-09-17 23:12:08.918901634 +0000 UTC m=+83.605974211"
Sep 17 23:12:09.047287 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:09.047159    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-bzzqr" podStartSLOduration=2.540158753 podStartE2EDuration="8.047134516s" totalImagesPullingTime="5.506975763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:08.980239175 +0000 UTC m=+83.667311750" watchObservedRunningTime="2026-09-17 23:12:09.047134516 +0000 UTC m=+83.734207092"
Sep 17 23:12:09.137624 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:09.137559    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-6786h" podStartSLOduration=2.501293765 podStartE2EDuration="8.137532561s" totalImagesPullingTime="5.636238796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:09.072672493 +0000 UTC m=+83.759745070" watchObservedRunningTime="2026-09-17 23:12:09.137532561 +0000 UTC m=+83.824605139"
Sep 17 23:12:09.211576 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:09.211518    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-qdg2c" podStartSLOduration=33.806493333 podStartE2EDuration="39.21149749s" totalImagesPullingTime="5.405004157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:09.139403521 +0000 UTC m=+83.826476097" watchObservedRunningTime="2026-09-17 23:12:09.21149749 +0000 UTC m=+83.898570071"
Sep 17 23:12:09.212160 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:09.212131    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-7jbxb" podStartSLOduration=2.575489724 podStartE2EDuration="8.212120331s" totalImagesPullingTime="5.636630607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:09.208847799 +0000 UTC m=+83.895920368" watchObservedRunningTime="2026-09-17 23:12:09.212120331 +0000 UTC m=+83.899192899"
Sep 17 23:12:09.820522 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:09.820476    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2sxs" event={"ID":"023d5e17-4668-4320-9949-9ec7f5a3feef","Type":"ContainerStarted","Data":"94beb1c66147d3ace07294b0b4ab6fcd7b0a22fbedf1461f7d3bb13c33047627"}
Sep 17 23:12:10.829852 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:10.829804    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2sxs" event={"ID":"023d5e17-4668-4320-9949-9ec7f5a3feef","Type":"ContainerStarted","Data":"3ee5b3a1c8466894b54cd20b6fe997ab6e9c0219eefef179b2b3b8e2e9e506e8"}
Sep 17 23:12:10.863472 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:10.863418    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f2sxs" podStartSLOduration=35.20333294 podStartE2EDuration="36.863400142s" totalImagesPullingTime="1.660067202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:10.862733385 +0000 UTC m=+85.549805961" watchObservedRunningTime="2026-09-17 23:12:10.863400142 +0000 UTC m=+85.550472719"
Sep 17 23:12:23.773463 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:23.773433    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5999d9d4c4-8hg92"
Sep 17 23:12:26.839268 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.839235    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kklrc"]
Sep 17 23:12:26.842506 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.842487    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:26.848334 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.848311    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 23:12:26.848655 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.848628    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tdlxl\""
Sep 17 23:12:26.848808 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.848667    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 23:12:26.849443 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.849425    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 23:12:26.849534 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.849481    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 23:12:26.958530 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.958490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6515a854-52fc-4271-ba77-f00f7e0dd25b-node-exporter-tls\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:26.958741 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.958555    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6515a854-52fc-4271-ba77-f00f7e0dd25b-root\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:26.958741 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.958582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6515a854-52fc-4271-ba77-f00f7e0dd25b-node-exporter-textfile\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:26.958741 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.958602    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6515a854-52fc-4271-ba77-f00f7e0dd25b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:26.958741 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.958624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6515a854-52fc-4271-ba77-f00f7e0dd25b-node-exporter-accelerators-collector-config\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:26.958741 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.958652    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq7cg\" (UniqueName: \"kubernetes.io/projected/6515a854-52fc-4271-ba77-f00f7e0dd25b-kube-api-access-hq7cg\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:26.958946 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.958743    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6515a854-52fc-4271-ba77-f00f7e0dd25b-sys\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:26.958946 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.958785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6515a854-52fc-4271-ba77-f00f7e0dd25b-node-exporter-wtmp\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:26.958946 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:26.958806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6515a854-52fc-4271-ba77-f00f7e0dd25b-metrics-client-ca\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.060221 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.060181    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6515a854-52fc-4271-ba77-f00f7e0dd25b-metrics-client-ca\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.060399 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.060236    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6515a854-52fc-4271-ba77-f00f7e0dd25b-node-exporter-tls\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.060399 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.060308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6515a854-52fc-4271-ba77-f00f7e0dd25b-root\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.060399 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.060352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6515a854-52fc-4271-ba77-f00f7e0dd25b-node-exporter-textfile\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.060399 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.060378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6515a854-52fc-4271-ba77-f00f7e0dd25b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.060615 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.060418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6515a854-52fc-4271-ba77-f00f7e0dd25b-node-exporter-accelerators-collector-config\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.060615 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.060449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq7cg\" (UniqueName: \"kubernetes.io/projected/6515a854-52fc-4271-ba77-f00f7e0dd25b-kube-api-access-hq7cg\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.060615 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.060491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6515a854-52fc-4271-ba77-f00f7e0dd25b-sys\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.060615 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.060532    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6515a854-52fc-4271-ba77-f00f7e0dd25b-node-exporter-wtmp\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.060824 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.060710    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6515a854-52fc-4271-ba77-f00f7e0dd25b-node-exporter-wtmp\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.060824 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.060743    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6515a854-52fc-4271-ba77-f00f7e0dd25b-node-exporter-textfile\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.060960 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.060912    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6515a854-52fc-4271-ba77-f00f7e0dd25b-metrics-client-ca\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.061173 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.061136    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6515a854-52fc-4271-ba77-f00f7e0dd25b-sys\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.061264 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.061161    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6515a854-52fc-4271-ba77-f00f7e0dd25b-root\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.061378 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.061357    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6515a854-52fc-4271-ba77-f00f7e0dd25b-node-exporter-accelerators-collector-config\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.063661 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.063616    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6515a854-52fc-4271-ba77-f00f7e0dd25b-node-exporter-tls\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.063954 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.063932    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6515a854-52fc-4271-ba77-f00f7e0dd25b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.070969 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.070945    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq7cg\" (UniqueName: \"kubernetes.io/projected/6515a854-52fc-4271-ba77-f00f7e0dd25b-kube-api-access-hq7cg\") pod \"node-exporter-kklrc\" (UID: \"6515a854-52fc-4271-ba77-f00f7e0dd25b\") " pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.152074 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.152041    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kklrc"
Sep 17 23:12:27.163429 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:12:27.163398    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6515a854_52fc_4271_ba77_f00f7e0dd25b.slice/crio-3281b928acf66418b39fdd0c9d62f0d1211009c1823c8ae3f00099668250c788 WatchSource:0}: Error finding container 3281b928acf66418b39fdd0c9d62f0d1211009c1823c8ae3f00099668250c788: Status 404 returned error can't find the container with id 3281b928acf66418b39fdd0c9d62f0d1211009c1823c8ae3f00099668250c788
Sep 17 23:12:27.894174 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:27.894138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kklrc" event={"ID":"6515a854-52fc-4271-ba77-f00f7e0dd25b","Type":"ContainerStarted","Data":"3281b928acf66418b39fdd0c9d62f0d1211009c1823c8ae3f00099668250c788"}
Sep 17 23:12:28.899232 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:28.899194    2582 generic.go:356] "Generic (PLEG): container finished" podID="6515a854-52fc-4271-ba77-f00f7e0dd25b" containerID="be16e342fb5ac58f4a37f49e5c9a1afc319f4eeeb82f2af7b60d7483164f728a" exitCode=0
Sep 17 23:12:28.899672 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:28.899294    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kklrc" event={"ID":"6515a854-52fc-4271-ba77-f00f7e0dd25b","Type":"ContainerDied","Data":"be16e342fb5ac58f4a37f49e5c9a1afc319f4eeeb82f2af7b60d7483164f728a"}
Sep 17 23:12:29.742600 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:29.742567    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5xnb7"
Sep 17 23:12:29.904924 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:29.904892    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kklrc" event={"ID":"6515a854-52fc-4271-ba77-f00f7e0dd25b","Type":"ContainerStarted","Data":"f3b45cfc0a506dc032769465e8f86214926b696e5e6250babda66690bbd02cc3"}
Sep 17 23:12:29.905312 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:29.904945    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kklrc" event={"ID":"6515a854-52fc-4271-ba77-f00f7e0dd25b","Type":"ContainerStarted","Data":"f58e9ccfd5cb0f2472bf1d8f28f165cd6cca9a0b70a2e9ce716586151b3de2bb"}
Sep 17 23:12:29.933755 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:29.933699    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kklrc" podStartSLOduration=2.914530783 podStartE2EDuration="3.933679678s" totalImagesPullingTime="949.571843ms" totalInitContainerRuntime="69.577052ms" isStatefulPod=true podCreationTimestamp="2026-09-17 23:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:29.932597654 +0000 UTC m=+104.619670247" watchObservedRunningTime="2026-09-17 23:12:29.933679678 +0000 UTC m=+104.620752256"
Sep 17 23:12:44.959300 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:44.959263    2582 generic.go:356] "Generic (PLEG): container finished" podID="2043c82f-26d1-4f3b-add6-385c35468db3" containerID="df8a3392739c4fd507a5ab3e689b89e432facb5dc348bfe36e00ebbc31ed4d64" exitCode=0
Sep 17 23:12:44.959711 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:44.959315    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9" event={"ID":"2043c82f-26d1-4f3b-add6-385c35468db3","Type":"ContainerDied","Data":"df8a3392739c4fd507a5ab3e689b89e432facb5dc348bfe36e00ebbc31ed4d64"}
Sep 17 23:12:44.959711 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:44.959622    2582 scope.go:117] "RemoveContainer" containerID="df8a3392739c4fd507a5ab3e689b89e432facb5dc348bfe36e00ebbc31ed4d64"
Sep 17 23:12:45.964734 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:12:45.964692    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-wvxp9" event={"ID":"2043c82f-26d1-4f3b-add6-385c35468db3","Type":"ContainerStarted","Data":"8855a5b3a1973d875a339ca92534132972c811c458fd7f27c440e8e1b50a1419"}
Sep 17 23:15:47.250481 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:15:47.250453    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 23:16:50.059068 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.059031    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-9wdxd"]
Sep 17 23:16:50.062295 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.062275    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-9wdxd"
Sep 17 23:16:50.066618 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.066594    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-fmvmx\""
Sep 17 23:16:50.066758 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.066633    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 23:16:50.066758 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.066636    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 23:16:50.067027 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.067009    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 23:16:50.075074 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.075051    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-9wdxd"]
Sep 17 23:16:50.221794 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.221753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fntm\" (UniqueName: \"kubernetes.io/projected/f344da7b-9c68-4d6f-910c-cc71effd4fe7-kube-api-access-4fntm\") pod \"seaweedfs-64568bcd49-9wdxd\" (UID: \"f344da7b-9c68-4d6f-910c-cc71effd4fe7\") " pod="kserve/seaweedfs-64568bcd49-9wdxd"
Sep 17 23:16:50.221794 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.221798    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f344da7b-9c68-4d6f-910c-cc71effd4fe7-data\") pod \"seaweedfs-64568bcd49-9wdxd\" (UID: \"f344da7b-9c68-4d6f-910c-cc71effd4fe7\") " pod="kserve/seaweedfs-64568bcd49-9wdxd"
Sep 17 23:16:50.323179 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.323097    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fntm\" (UniqueName: \"kubernetes.io/projected/f344da7b-9c68-4d6f-910c-cc71effd4fe7-kube-api-access-4fntm\") pod \"seaweedfs-64568bcd49-9wdxd\" (UID: \"f344da7b-9c68-4d6f-910c-cc71effd4fe7\") " pod="kserve/seaweedfs-64568bcd49-9wdxd"
Sep 17 23:16:50.323179 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.323138    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f344da7b-9c68-4d6f-910c-cc71effd4fe7-data\") pod \"seaweedfs-64568bcd49-9wdxd\" (UID: \"f344da7b-9c68-4d6f-910c-cc71effd4fe7\") " pod="kserve/seaweedfs-64568bcd49-9wdxd"
Sep 17 23:16:50.323495 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.323480    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f344da7b-9c68-4d6f-910c-cc71effd4fe7-data\") pod \"seaweedfs-64568bcd49-9wdxd\" (UID: \"f344da7b-9c68-4d6f-910c-cc71effd4fe7\") " pod="kserve/seaweedfs-64568bcd49-9wdxd"
Sep 17 23:16:50.339151 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.339127    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fntm\" (UniqueName: \"kubernetes.io/projected/f344da7b-9c68-4d6f-910c-cc71effd4fe7-kube-api-access-4fntm\") pod \"seaweedfs-64568bcd49-9wdxd\" (UID: \"f344da7b-9c68-4d6f-910c-cc71effd4fe7\") " pod="kserve/seaweedfs-64568bcd49-9wdxd"
Sep 17 23:16:50.372597 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.372525    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-9wdxd"
Sep 17 23:16:50.659569 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.659537    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-9wdxd"]
Sep 17 23:16:50.662972 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:16:50.662934    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf344da7b_9c68_4d6f_910c_cc71effd4fe7.slice/crio-9c623e96291db1298191f0d604dce49412c3d9af7b965cb4513a7d04533a3dc1 WatchSource:0}: Error finding container 9c623e96291db1298191f0d604dce49412c3d9af7b965cb4513a7d04533a3dc1: Status 404 returned error can't find the container with id 9c623e96291db1298191f0d604dce49412c3d9af7b965cb4513a7d04533a3dc1
Sep 17 23:16:50.664629 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.664607    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:16:50.836098 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:50.836060    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-9wdxd" event={"ID":"f344da7b-9c68-4d6f-910c-cc71effd4fe7","Type":"ContainerStarted","Data":"9c623e96291db1298191f0d604dce49412c3d9af7b965cb4513a7d04533a3dc1"}
Sep 17 23:16:54.852673 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:54.852629    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-9wdxd" event={"ID":"f344da7b-9c68-4d6f-910c-cc71effd4fe7","Type":"ContainerStarted","Data":"d850f378985025334a5d146704e13234ff0e702ce4a3aae1e30a5c6eb085e25d"}
Sep 17 23:16:54.853180 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:54.852763    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-9wdxd"
Sep 17 23:16:54.874739 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:16:54.874685    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-9wdxd" podStartSLOduration=1.631817507 podStartE2EDuration="4.874666516s" totalImagesPullingTime="3.242849009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:54.872770121 +0000 UTC m=+369.559842698" watchObservedRunningTime="2026-09-17 23:16:54.874666516 +0000 UTC m=+369.561739101"
Sep 17 23:17:00.858293 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:17:00.858258    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-9wdxd"
Sep 17 23:18:07.889957 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:07.889879    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-bk44s"]
Sep 17 23:18:07.892965 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:07.892947    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-bk44s"
Sep 17 23:18:07.895640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:07.895621    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-7qv4x\""
Sep 17 23:18:07.895938 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:07.895923    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 23:18:07.901634 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:07.901614    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-qh6x4"]
Sep 17 23:18:07.904375 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:07.904353    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-qh6x4"
Sep 17 23:18:07.905941 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:07.905921    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-bk44s"]
Sep 17 23:18:07.907721 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:07.907697    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 23:18:07.908204 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:07.908189    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-s297m\""
Sep 17 23:18:07.908509 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:07.908495    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 23:18:07.918866 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:07.918822    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-qh6x4"]
Sep 17 23:18:07.968029 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:07.968002    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a6a9bd-6479-47cc-bb74-c081b63949a9-tls-certs\") pod \"model-serving-api-7b954ffc95-bk44s\" (UID: \"f8a6a9bd-6479-47cc-bb74-c081b63949a9\") " pod="kserve/model-serving-api-7b954ffc95-bk44s"
Sep 17 23:18:07.968136 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:07.968062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz4mh\" (UniqueName: \"kubernetes.io/projected/f8a6a9bd-6479-47cc-bb74-c081b63949a9-kube-api-access-cz4mh\") pod \"model-serving-api-7b954ffc95-bk44s\" (UID: \"f8a6a9bd-6479-47cc-bb74-c081b63949a9\") " pod="kserve/model-serving-api-7b954ffc95-bk44s"
Sep 17 23:18:08.068693 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.068665    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7a8953b-aa7d-43e3-8823-4e581c5f716d-cert\") pod \"odh-model-controller-7b966d7984-qh6x4\" (UID: \"a7a8953b-aa7d-43e3-8823-4e581c5f716d\") " pod="kserve/odh-model-controller-7b966d7984-qh6x4"
Sep 17 23:18:08.068818 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.068700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz4mh\" (UniqueName: \"kubernetes.io/projected/f8a6a9bd-6479-47cc-bb74-c081b63949a9-kube-api-access-cz4mh\") pod \"model-serving-api-7b954ffc95-bk44s\" (UID: \"f8a6a9bd-6479-47cc-bb74-c081b63949a9\") " pod="kserve/model-serving-api-7b954ffc95-bk44s"
Sep 17 23:18:08.068818 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.068725    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkrqt\" (UniqueName: \"kubernetes.io/projected/a7a8953b-aa7d-43e3-8823-4e581c5f716d-kube-api-access-hkrqt\") pod \"odh-model-controller-7b966d7984-qh6x4\" (UID: \"a7a8953b-aa7d-43e3-8823-4e581c5f716d\") " pod="kserve/odh-model-controller-7b966d7984-qh6x4"
Sep 17 23:18:08.068818 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.068748    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a6a9bd-6479-47cc-bb74-c081b63949a9-tls-certs\") pod \"model-serving-api-7b954ffc95-bk44s\" (UID: \"f8a6a9bd-6479-47cc-bb74-c081b63949a9\") " pod="kserve/model-serving-api-7b954ffc95-bk44s"
Sep 17 23:18:08.068981 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.068926    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7a8953b-aa7d-43e3-8823-4e581c5f716d-metrics-certs\") pod \"odh-model-controller-7b966d7984-qh6x4\" (UID: \"a7a8953b-aa7d-43e3-8823-4e581c5f716d\") " pod="kserve/odh-model-controller-7b966d7984-qh6x4"
Sep 17 23:18:08.071347 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.071322    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a6a9bd-6479-47cc-bb74-c081b63949a9-tls-certs\") pod \"model-serving-api-7b954ffc95-bk44s\" (UID: \"f8a6a9bd-6479-47cc-bb74-c081b63949a9\") " pod="kserve/model-serving-api-7b954ffc95-bk44s"
Sep 17 23:18:08.082630 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.082609    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz4mh\" (UniqueName: \"kubernetes.io/projected/f8a6a9bd-6479-47cc-bb74-c081b63949a9-kube-api-access-cz4mh\") pod \"model-serving-api-7b954ffc95-bk44s\" (UID: \"f8a6a9bd-6479-47cc-bb74-c081b63949a9\") " pod="kserve/model-serving-api-7b954ffc95-bk44s"
Sep 17 23:18:08.169483 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.169416    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7a8953b-aa7d-43e3-8823-4e581c5f716d-cert\") pod \"odh-model-controller-7b966d7984-qh6x4\" (UID: \"a7a8953b-aa7d-43e3-8823-4e581c5f716d\") " pod="kserve/odh-model-controller-7b966d7984-qh6x4"
Sep 17 23:18:08.169483 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.169453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkrqt\" (UniqueName: \"kubernetes.io/projected/a7a8953b-aa7d-43e3-8823-4e581c5f716d-kube-api-access-hkrqt\") pod \"odh-model-controller-7b966d7984-qh6x4\" (UID: \"a7a8953b-aa7d-43e3-8823-4e581c5f716d\") " pod="kserve/odh-model-controller-7b966d7984-qh6x4"
Sep 17 23:18:08.169483 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.169476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7a8953b-aa7d-43e3-8823-4e581c5f716d-metrics-certs\") pod \"odh-model-controller-7b966d7984-qh6x4\" (UID: \"a7a8953b-aa7d-43e3-8823-4e581c5f716d\") " pod="kserve/odh-model-controller-7b966d7984-qh6x4"
Sep 17 23:18:08.171880 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.171856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7a8953b-aa7d-43e3-8823-4e581c5f716d-metrics-certs\") pod \"odh-model-controller-7b966d7984-qh6x4\" (UID: \"a7a8953b-aa7d-43e3-8823-4e581c5f716d\") " pod="kserve/odh-model-controller-7b966d7984-qh6x4"
Sep 17 23:18:08.171996 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.171979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7a8953b-aa7d-43e3-8823-4e581c5f716d-cert\") pod \"odh-model-controller-7b966d7984-qh6x4\" (UID: \"a7a8953b-aa7d-43e3-8823-4e581c5f716d\") " pod="kserve/odh-model-controller-7b966d7984-qh6x4"
Sep 17 23:18:08.190115 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.190091    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkrqt\" (UniqueName: \"kubernetes.io/projected/a7a8953b-aa7d-43e3-8823-4e581c5f716d-kube-api-access-hkrqt\") pod \"odh-model-controller-7b966d7984-qh6x4\" (UID: \"a7a8953b-aa7d-43e3-8823-4e581c5f716d\") " pod="kserve/odh-model-controller-7b966d7984-qh6x4"
Sep 17 23:18:08.203931 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.203908    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-bk44s"
Sep 17 23:18:08.215015 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.214998    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-qh6x4"
Sep 17 23:18:08.479272 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.479249    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-bk44s"]
Sep 17 23:18:08.481793 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:18:08.481765    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a6a9bd_6479_47cc_bb74_c081b63949a9.slice/crio-5db299c0c12d2ff5ff0d8cc4d6fc8bfe76a421ee7ba242b1631fdf88928242fc WatchSource:0}: Error finding container 5db299c0c12d2ff5ff0d8cc4d6fc8bfe76a421ee7ba242b1631fdf88928242fc: Status 404 returned error can't find the container with id 5db299c0c12d2ff5ff0d8cc4d6fc8bfe76a421ee7ba242b1631fdf88928242fc
Sep 17 23:18:08.500334 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:08.500311    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-qh6x4"]
Sep 17 23:18:08.501403 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:18:08.501380    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a8953b_aa7d_43e3_8823_4e581c5f716d.slice/crio-fc7cd3b57bd3e7740503c06b0ed0e5445071b1230d9e048595494987b69a335c WatchSource:0}: Error finding container fc7cd3b57bd3e7740503c06b0ed0e5445071b1230d9e048595494987b69a335c: Status 404 returned error can't find the container with id fc7cd3b57bd3e7740503c06b0ed0e5445071b1230d9e048595494987b69a335c
Sep 17 23:18:09.111900 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:09.111859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-qh6x4" event={"ID":"a7a8953b-aa7d-43e3-8823-4e581c5f716d","Type":"ContainerStarted","Data":"fc7cd3b57bd3e7740503c06b0ed0e5445071b1230d9e048595494987b69a335c"}
Sep 17 23:18:09.113511 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:09.113463    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-bk44s" event={"ID":"f8a6a9bd-6479-47cc-bb74-c081b63949a9","Type":"ContainerStarted","Data":"5db299c0c12d2ff5ff0d8cc4d6fc8bfe76a421ee7ba242b1631fdf88928242fc"}
Sep 17 23:18:13.131615 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:13.131576    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-qh6x4" event={"ID":"a7a8953b-aa7d-43e3-8823-4e581c5f716d","Type":"ContainerStarted","Data":"b5fdfc18e4801f88c1a10de170dea1205cb293f92fe45f44b8c9d56ab46fd072"}
Sep 17 23:18:13.132086 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:13.131674    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-qh6x4"
Sep 17 23:18:13.133091 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:13.133070    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-bk44s" event={"ID":"f8a6a9bd-6479-47cc-bb74-c081b63949a9","Type":"ContainerStarted","Data":"5a4407fb9838968b00472231f641c4930900ad4030da83b7c6516b0a1a566cac"}
Sep 17 23:18:13.133219 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:13.133122    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-bk44s"
Sep 17 23:18:13.153018 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:13.152982    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-qh6x4" podStartSLOduration=2.399885184 podStartE2EDuration="6.152971475s" totalImagesPullingTime="3.753086291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:13.15095533 +0000 UTC m=+447.838027906" watchObservedRunningTime="2026-09-17 23:18:13.152971475 +0000 UTC m=+447.840044052"
Sep 17 23:18:13.168951 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:13.168906    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-bk44s" podStartSLOduration=2.444561735 podStartE2EDuration="6.1688901s" totalImagesPullingTime="3.724328365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:13.167817156 +0000 UTC m=+447.854889734" watchObservedRunningTime="2026-09-17 23:18:13.1688901 +0000 UTC m=+447.855962677"
Sep 17 23:18:24.138560 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:24.138526    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-qh6x4"
Sep 17 23:18:24.140476 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:18:24.140452    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-bk44s"
Sep 17 23:19:56.309910 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.309871    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"]
Sep 17 23:19:56.313187 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.313170    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.316023 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.315999    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:19:56.316162 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.316051    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:19:56.316236 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.316104    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:19:56.316291 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.316255    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 23:19:56.316484 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.316458    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f2cbg\""
Sep 17 23:19:56.316615 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.316509    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 23:19:56.317122 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.317107    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:19:56.326888 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.326868    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"]
Sep 17 23:19:56.354292 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.354270    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d19dd52-dc0f-48d8-b26d-0d6018002af1-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.354392 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.354318    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.354432 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.354378    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d19dd52-dc0f-48d8-b26d-0d6018002af1-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.354466 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.354429    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvrj2\" (UniqueName: \"kubernetes.io/projected/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kube-api-access-kvrj2\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.354531 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.354515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.400125 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.400099    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"]
Sep 17 23:19:56.403230 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.403214    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.405822 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.405804    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 23:19:56.415573 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.415552    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"]
Sep 17 23:19:56.454822 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.454789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.454822 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.454821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-t9fj5\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.455040 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.454864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d19dd52-dc0f-48d8-b26d-0d6018002af1-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.455040 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:19:56.454964    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 23:19:56.455040 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.454991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-t9fj5\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.455040 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:19:56.455020    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d19dd52-dc0f-48d8-b26d-0d6018002af1-proxy-tls podName:6d19dd52-dc0f-48d8-b26d-0d6018002af1 nodeName:}" failed. No retries permitted until 2026-09-17 23:19:56.955006483 +0000 UTC m=+551.642079038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6d19dd52-dc0f-48d8-b26d-0d6018002af1-proxy-tls") pod "isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" (UID: "6d19dd52-dc0f-48d8-b26d-0d6018002af1") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 23:19:56.455241 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.455046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.455241 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.455068    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d19dd52-dc0f-48d8-b26d-0d6018002af1-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.455241 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.455128    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvrj2\" (UniqueName: \"kubernetes.io/projected/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kube-api-access-kvrj2\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.455241 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.455171    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-t9fj5\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.455241 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.455212    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-t9fj5\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.455241 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.455232    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.455511 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.455277    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpwr5\" (UniqueName: \"kubernetes.io/projected/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kube-api-access-bpwr5\") pod \"isvc-canary-rollback-predictor-7df6589c46-t9fj5\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.455689 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.455672    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.455826 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.455808    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d19dd52-dc0f-48d8-b26d-0d6018002af1-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.464658 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.464635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvrj2\" (UniqueName: \"kubernetes.io/projected/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kube-api-access-kvrj2\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.556126 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.556089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-t9fj5\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.556305 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.556134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-t9fj5\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.556305 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.556155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-t9fj5\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.556305 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.556173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwr5\" (UniqueName: \"kubernetes.io/projected/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kube-api-access-bpwr5\") pod \"isvc-canary-rollback-predictor-7df6589c46-t9fj5\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.556305 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.556228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-t9fj5\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.556571 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.556541    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-t9fj5\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.556868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.556821    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-t9fj5\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.556966 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.556944    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-t9fj5\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.558743 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.558722    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-t9fj5\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.571141 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.571093    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpwr5\" (UniqueName: \"kubernetes.io/projected/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kube-api-access-bpwr5\") pod \"isvc-canary-rollback-predictor-7df6589c46-t9fj5\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.714493 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.714459    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:19:56.959385 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.959347    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d19dd52-dc0f-48d8-b26d-0d6018002af1-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.961931 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.961912    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d19dd52-dc0f-48d8-b26d-0d6018002af1-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:56.987941 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:56.987912    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"]
Sep 17 23:19:56.990454 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:19:56.990430    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7d81de_1b3e_4fa0_93dc_02e306b7b673.slice/crio-c1f6e3833005b48c3cc4010cac896e4cf08cfb124be513125b0e2a5f429390e0 WatchSource:0}: Error finding container c1f6e3833005b48c3cc4010cac896e4cf08cfb124be513125b0e2a5f429390e0: Status 404 returned error can't find the container with id c1f6e3833005b48c3cc4010cac896e4cf08cfb124be513125b0e2a5f429390e0
Sep 17 23:19:57.225054 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:57.224972    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:19:57.500613 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:57.500575    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"]
Sep 17 23:19:57.503465 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:19:57.503439    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d19dd52_dc0f_48d8_b26d_0d6018002af1.slice/crio-653fc9934fc98555abc6fe34f08ab78ad321e58f2e2e75df1f26af566b5a00d3 WatchSource:0}: Error finding container 653fc9934fc98555abc6fe34f08ab78ad321e58f2e2e75df1f26af566b5a00d3: Status 404 returned error can't find the container with id 653fc9934fc98555abc6fe34f08ab78ad321e58f2e2e75df1f26af566b5a00d3
Sep 17 23:19:57.515141 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:57.515116    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" event={"ID":"0b7d81de-1b3e-4fa0-93dc-02e306b7b673","Type":"ContainerStarted","Data":"c1f6e3833005b48c3cc4010cac896e4cf08cfb124be513125b0e2a5f429390e0"}
Sep 17 23:19:57.516473 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:19:57.516449    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" event={"ID":"6d19dd52-dc0f-48d8-b26d-0d6018002af1","Type":"ContainerStarted","Data":"653fc9934fc98555abc6fe34f08ab78ad321e58f2e2e75df1f26af566b5a00d3"}
Sep 17 23:20:01.538356 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:01.538314    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" event={"ID":"6d19dd52-dc0f-48d8-b26d-0d6018002af1","Type":"ContainerStarted","Data":"d42bacbe144e75cd11f0b997366bb02f1384b82d9dc7d0d1d2089066c84e7c49"}
Sep 17 23:20:01.540179 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:01.540145    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" event={"ID":"0b7d81de-1b3e-4fa0-93dc-02e306b7b673","Type":"ContainerStarted","Data":"9b4b1c13520feb29bcc058edd9ac76aad1e66ca7cc1629b210b8c61aa3aa1752"}
Sep 17 23:20:05.559642 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:05.559608    2582 generic.go:356] "Generic (PLEG): container finished" podID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerID="9b4b1c13520feb29bcc058edd9ac76aad1e66ca7cc1629b210b8c61aa3aa1752" exitCode=0
Sep 17 23:20:05.560179 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:05.559686    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" event={"ID":"0b7d81de-1b3e-4fa0-93dc-02e306b7b673","Type":"ContainerDied","Data":"9b4b1c13520feb29bcc058edd9ac76aad1e66ca7cc1629b210b8c61aa3aa1752"}
Sep 17 23:20:05.561391 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:05.561366    2582 generic.go:356] "Generic (PLEG): container finished" podID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerID="d42bacbe144e75cd11f0b997366bb02f1384b82d9dc7d0d1d2089066c84e7c49" exitCode=0
Sep 17 23:20:05.561529 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:05.561396    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" event={"ID":"6d19dd52-dc0f-48d8-b26d-0d6018002af1","Type":"ContainerDied","Data":"d42bacbe144e75cd11f0b997366bb02f1384b82d9dc7d0d1d2089066c84e7c49"}
Sep 17 23:20:22.672174 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:22.672064    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" event={"ID":"6d19dd52-dc0f-48d8-b26d-0d6018002af1","Type":"ContainerStarted","Data":"d4f9174695686c24c933d4be7aeb504b561321a0cd942124808e0ae44dc3108d"}
Sep 17 23:20:22.674903 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:22.674794    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" event={"ID":"0b7d81de-1b3e-4fa0-93dc-02e306b7b673","Type":"ContainerStarted","Data":"5f38c588596eb6617c1d9050dcb2baaa396744e763220a37a88d8d08579dd11a"}
Sep 17 23:20:24.684570 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:24.684529    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" event={"ID":"6d19dd52-dc0f-48d8-b26d-0d6018002af1","Type":"ContainerStarted","Data":"a6e09e52f011a61384c93513118fb646c3377145d9ee9f4327087ed70cfb9fa2"}
Sep 17 23:20:24.685033 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:24.684645    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:20:24.685033 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:24.684866    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:20:24.686432 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:24.686396    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 23:20:24.686907 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:24.686881    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" event={"ID":"0b7d81de-1b3e-4fa0-93dc-02e306b7b673","Type":"ContainerStarted","Data":"9a35adf61b50460279dc92837a279027bf599497e32967173961314251d0284f"}
Sep 17 23:20:24.687110 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:24.687093    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:20:24.687170 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:24.687145    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:20:24.688124 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:24.688104    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 23:20:24.706112 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:24.706063    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" podStartSLOduration=2.839264919 podStartE2EDuration="28.706046546s" totalImagesPullingTime="21.905948906s" totalInitContainerRuntime="3.960832721s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:24.704877007 +0000 UTC m=+579.391949584" watchObservedRunningTime="2026-09-17 23:20:24.706046546 +0000 UTC m=+579.393119124"
Sep 17 23:20:24.723513 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:24.723473    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" podStartSLOduration=2.501618889 podStartE2EDuration="28.723461086s" totalImagesPullingTime="22.422719239s" totalInitContainerRuntime="3.799122958s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:24.722087198 +0000 UTC m=+579.409159780" watchObservedRunningTime="2026-09-17 23:20:24.723461086 +0000 UTC m=+579.410533662"
Sep 17 23:20:25.694095 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:25.694037    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 23:20:25.694540 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:25.694329    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 23:20:30.698289 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:30.698256    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:20:30.698877 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:30.698736    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:20:30.698877 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:30.698793    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 23:20:30.699181 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:30.699160    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 23:20:40.699720 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:40.699684    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 23:20:40.700133 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:40.699683    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 23:20:50.699005 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:50.698966    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 23:20:50.699387 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:20:50.699060    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 23:21:00.699280 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:00.699194    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 23:21:00.699730 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:00.699189    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 23:21:10.699240 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:10.699195    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 23:21:10.699240 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:10.699209    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 23:21:20.699526 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:20.699489    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 23:21:20.699968 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:20.699493    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 23:21:30.699100 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:30.699057    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 17 23:21:30.699560 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:30.699064    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 17 23:21:40.699006 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:40.698968    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:21:40.701204 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:40.699723    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:21:40.845491 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:40.845456    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"]
Sep 17 23:21:40.970550 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:40.970457    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kserve-container" containerID="cri-o://5f38c588596eb6617c1d9050dcb2baaa396744e763220a37a88d8d08579dd11a" gracePeriod=30
Sep 17 23:21:40.970550 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:40.970493    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kube-rbac-proxy" containerID="cri-o://9a35adf61b50460279dc92837a279027bf599497e32967173961314251d0284f" gracePeriod=30
Sep 17 23:21:41.977294 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:41.977258    2582 generic.go:356] "Generic (PLEG): container finished" podID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerID="9a35adf61b50460279dc92837a279027bf599497e32967173961314251d0284f" exitCode=2
Sep 17 23:21:41.977724 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:41.977330    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" event={"ID":"0b7d81de-1b3e-4fa0-93dc-02e306b7b673","Type":"ContainerDied","Data":"9a35adf61b50460279dc92837a279027bf599497e32967173961314251d0284f"}
Sep 17 23:21:44.992149 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:44.992121    2582 generic.go:356] "Generic (PLEG): container finished" podID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerID="5f38c588596eb6617c1d9050dcb2baaa396744e763220a37a88d8d08579dd11a" exitCode=0
Sep 17 23:21:44.992466 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:44.992195    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" event={"ID":"0b7d81de-1b3e-4fa0-93dc-02e306b7b673","Type":"ContainerDied","Data":"5f38c588596eb6617c1d9050dcb2baaa396744e763220a37a88d8d08579dd11a"}
Sep 17 23:21:45.109815 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.109793    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:21:45.170187 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.170150    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-cabundle-cert\") pod \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") "
Sep 17 23:21:45.170352 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.170214    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-proxy-tls\") pod \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") "
Sep 17 23:21:45.170352 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.170232    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpwr5\" (UniqueName: \"kubernetes.io/projected/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kube-api-access-bpwr5\") pod \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") "
Sep 17 23:21:45.170352 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.170271    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kube-rbac-proxy-sar-config\") pod \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") "
Sep 17 23:21:45.170352 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.170304    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kserve-provision-location\") pod \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\" (UID: \"0b7d81de-1b3e-4fa0-93dc-02e306b7b673\") "
Sep 17 23:21:45.170662 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.170628    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b7d81de-1b3e-4fa0-93dc-02e306b7b673" (UID: "0b7d81de-1b3e-4fa0-93dc-02e306b7b673"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:21:45.170662 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.170638    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b7d81de-1b3e-4fa0-93dc-02e306b7b673" (UID: "0b7d81de-1b3e-4fa0-93dc-02e306b7b673"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:21:45.170827 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.170703    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b7d81de-1b3e-4fa0-93dc-02e306b7b673" (UID: "0b7d81de-1b3e-4fa0-93dc-02e306b7b673"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:21:45.172750 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.172727    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kube-api-access-bpwr5" (OuterVolumeSpecName: "kube-api-access-bpwr5") pod "0b7d81de-1b3e-4fa0-93dc-02e306b7b673" (UID: "0b7d81de-1b3e-4fa0-93dc-02e306b7b673"). InnerVolumeSpecName "kube-api-access-bpwr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:21:45.172860 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.172791    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b7d81de-1b3e-4fa0-93dc-02e306b7b673" (UID: "0b7d81de-1b3e-4fa0-93dc-02e306b7b673"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:21:45.271617 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.271576    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:21:45.271617 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.271613    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:21:45.271617 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.271622    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:21:45.271617 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.271630    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpwr5\" (UniqueName: \"kubernetes.io/projected/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kube-api-access-bpwr5\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:21:45.271930 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.271640    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b7d81de-1b3e-4fa0-93dc-02e306b7b673-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:21:45.998239 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.998205    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5" event={"ID":"0b7d81de-1b3e-4fa0-93dc-02e306b7b673","Type":"ContainerDied","Data":"c1f6e3833005b48c3cc4010cac896e4cf08cfb124be513125b0e2a5f429390e0"}
Sep 17 23:21:45.998239 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.998229    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"
Sep 17 23:21:45.998637 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:45.998292    2582 scope.go:117] "RemoveContainer" containerID="9a35adf61b50460279dc92837a279027bf599497e32967173961314251d0284f"
Sep 17 23:21:46.006828 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.006811    2582 scope.go:117] "RemoveContainer" containerID="5f38c588596eb6617c1d9050dcb2baaa396744e763220a37a88d8d08579dd11a"
Sep 17 23:21:46.014200 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.014157    2582 scope.go:117] "RemoveContainer" containerID="9b4b1c13520feb29bcc058edd9ac76aad1e66ca7cc1629b210b8c61aa3aa1752"
Sep 17 23:21:46.019851 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.019809    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"]
Sep 17 23:21:46.024359 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.024335    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-t9fj5"]
Sep 17 23:21:46.035749 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.035727    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"]
Sep 17 23:21:46.036022 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.035963    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kserve-container" containerID="cri-o://d4f9174695686c24c933d4be7aeb504b561321a0cd942124808e0ae44dc3108d" gracePeriod=30
Sep 17 23:21:46.036133 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.035993    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kube-rbac-proxy" containerID="cri-o://a6e09e52f011a61384c93513118fb646c3377145d9ee9f4327087ed70cfb9fa2" gracePeriod=30
Sep 17 23:21:46.113739 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.113711    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"]
Sep 17 23:21:46.114008 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.113996    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kube-rbac-proxy"
Sep 17 23:21:46.114053 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.114010    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kube-rbac-proxy"
Sep 17 23:21:46.114053 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.114020    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="storage-initializer"
Sep 17 23:21:46.114053 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.114026    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="storage-initializer"
Sep 17 23:21:46.114053 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.114032    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kserve-container"
Sep 17 23:21:46.114053 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.114038    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kserve-container"
Sep 17 23:21:46.114195 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.114095    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kube-rbac-proxy"
Sep 17 23:21:46.114195 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.114105    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" containerName="kserve-container"
Sep 17 23:21:46.118447 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.118431    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.120870 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.120851    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 23:21:46.128447 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.128425    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"]
Sep 17 23:21:46.179051 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.179031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e199634a-3078-4ca0-819d-b2714883748b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-zw6z6\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.179124 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.179063    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e199634a-3078-4ca0-819d-b2714883748b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-zw6z6\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.179175 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.179124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e199634a-3078-4ca0-819d-b2714883748b-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-zw6z6\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.179175 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.179153    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e199634a-3078-4ca0-819d-b2714883748b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-zw6z6\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.179254 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.179175    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78vg5\" (UniqueName: \"kubernetes.io/projected/e199634a-3078-4ca0-819d-b2714883748b-kube-api-access-78vg5\") pod \"isvc-canary-rollback-predictor-7df6589c46-zw6z6\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.279783 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.279752    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e199634a-3078-4ca0-819d-b2714883748b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-zw6z6\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.279901 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.279788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e199634a-3078-4ca0-819d-b2714883748b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-zw6z6\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.279901 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.279816    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e199634a-3078-4ca0-819d-b2714883748b-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-zw6z6\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.279901 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.279858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e199634a-3078-4ca0-819d-b2714883748b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-zw6z6\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.279901 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.279890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78vg5\" (UniqueName: \"kubernetes.io/projected/e199634a-3078-4ca0-819d-b2714883748b-kube-api-access-78vg5\") pod \"isvc-canary-rollback-predictor-7df6589c46-zw6z6\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.280236 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.280200    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e199634a-3078-4ca0-819d-b2714883748b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-zw6z6\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.280556 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.280480    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e199634a-3078-4ca0-819d-b2714883748b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-zw6z6\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.280556 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.280480    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e199634a-3078-4ca0-819d-b2714883748b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-zw6z6\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.282275 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.282255    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e199634a-3078-4ca0-819d-b2714883748b-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-zw6z6\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.288407 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.288389    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78vg5\" (UniqueName: \"kubernetes.io/projected/e199634a-3078-4ca0-819d-b2714883748b-kube-api-access-78vg5\") pod \"isvc-canary-rollback-predictor-7df6589c46-zw6z6\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.428485 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.428436    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:46.701959 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:46.701798    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"]
Sep 17 23:21:46.704471 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:21:46.704443    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode199634a_3078_4ca0_819d_b2714883748b.slice/crio-1cf11ac20e518b53f5a2b04872bfe2939fed4aa6439aaf7456e2cb53ebb960ed WatchSource:0}: Error finding container 1cf11ac20e518b53f5a2b04872bfe2939fed4aa6439aaf7456e2cb53ebb960ed: Status 404 returned error can't find the container with id 1cf11ac20e518b53f5a2b04872bfe2939fed4aa6439aaf7456e2cb53ebb960ed
Sep 17 23:21:47.004895 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:47.004789    2582 generic.go:356] "Generic (PLEG): container finished" podID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerID="a6e09e52f011a61384c93513118fb646c3377145d9ee9f4327087ed70cfb9fa2" exitCode=2
Sep 17 23:21:47.004895 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:47.004867    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" event={"ID":"6d19dd52-dc0f-48d8-b26d-0d6018002af1","Type":"ContainerDied","Data":"a6e09e52f011a61384c93513118fb646c3377145d9ee9f4327087ed70cfb9fa2"}
Sep 17 23:21:47.006432 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:47.006409    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" event={"ID":"e199634a-3078-4ca0-819d-b2714883748b","Type":"ContainerStarted","Data":"af8fd5ccf292d5766d030d9a8d95822764df87424309a41e575ea6cebbe7e453"}
Sep 17 23:21:47.006548 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:47.006453    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" event={"ID":"e199634a-3078-4ca0-819d-b2714883748b","Type":"ContainerStarted","Data":"1cf11ac20e518b53f5a2b04872bfe2939fed4aa6439aaf7456e2cb53ebb960ed"}
Sep 17 23:21:47.396220 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:47.396179    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b7d81de-1b3e-4fa0-93dc-02e306b7b673" path="/var/lib/kubelet/pods/0b7d81de-1b3e-4fa0-93dc-02e306b7b673/volumes"
Sep 17 23:21:50.448580 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.448520    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:21:50.512301 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.512274    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvrj2\" (UniqueName: \"kubernetes.io/projected/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kube-api-access-kvrj2\") pod \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") "
Sep 17 23:21:50.512301 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.512302    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d19dd52-dc0f-48d8-b26d-0d6018002af1-cabundle-cert\") pod \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") "
Sep 17 23:21:50.512487 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.512344    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kube-rbac-proxy-sar-config\") pod \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") "
Sep 17 23:21:50.512487 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.512456    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kserve-provision-location\") pod \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") "
Sep 17 23:21:50.512598 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.512491    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d19dd52-dc0f-48d8-b26d-0d6018002af1-proxy-tls\") pod \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\" (UID: \"6d19dd52-dc0f-48d8-b26d-0d6018002af1\") "
Sep 17 23:21:50.512655 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.512631    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d19dd52-dc0f-48d8-b26d-0d6018002af1" (UID: "6d19dd52-dc0f-48d8-b26d-0d6018002af1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:21:50.512766 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.512744    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d19dd52-dc0f-48d8-b26d-0d6018002af1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6d19dd52-dc0f-48d8-b26d-0d6018002af1" (UID: "6d19dd52-dc0f-48d8-b26d-0d6018002af1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:21:50.512813 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.512774    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d19dd52-dc0f-48d8-b26d-0d6018002af1" (UID: "6d19dd52-dc0f-48d8-b26d-0d6018002af1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:21:50.514496 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.514476    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kube-api-access-kvrj2" (OuterVolumeSpecName: "kube-api-access-kvrj2") pod "6d19dd52-dc0f-48d8-b26d-0d6018002af1" (UID: "6d19dd52-dc0f-48d8-b26d-0d6018002af1"). InnerVolumeSpecName "kube-api-access-kvrj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:21:50.514594 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.514571    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d19dd52-dc0f-48d8-b26d-0d6018002af1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d19dd52-dc0f-48d8-b26d-0d6018002af1" (UID: "6d19dd52-dc0f-48d8-b26d-0d6018002af1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:21:50.613596 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.613571    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d19dd52-dc0f-48d8-b26d-0d6018002af1-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:21:50.613596 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.613592    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:21:50.613722 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.613602    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:21:50.613722 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.613611    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d19dd52-dc0f-48d8-b26d-0d6018002af1-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:21:50.613722 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:50.613620    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvrj2\" (UniqueName: \"kubernetes.io/projected/6d19dd52-dc0f-48d8-b26d-0d6018002af1-kube-api-access-kvrj2\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:21:51.024892 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.024857    2582 generic.go:356] "Generic (PLEG): container finished" podID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerID="d4f9174695686c24c933d4be7aeb504b561321a0cd942124808e0ae44dc3108d" exitCode=0
Sep 17 23:21:51.025058 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.024926    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"
Sep 17 23:21:51.025058 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.024944    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" event={"ID":"6d19dd52-dc0f-48d8-b26d-0d6018002af1","Type":"ContainerDied","Data":"d4f9174695686c24c933d4be7aeb504b561321a0cd942124808e0ae44dc3108d"}
Sep 17 23:21:51.025058 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.025015    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8" event={"ID":"6d19dd52-dc0f-48d8-b26d-0d6018002af1","Type":"ContainerDied","Data":"653fc9934fc98555abc6fe34f08ab78ad321e58f2e2e75df1f26af566b5a00d3"}
Sep 17 23:21:51.025204 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.025062    2582 scope.go:117] "RemoveContainer" containerID="a6e09e52f011a61384c93513118fb646c3377145d9ee9f4327087ed70cfb9fa2"
Sep 17 23:21:51.026954 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.026931    2582 generic.go:356] "Generic (PLEG): container finished" podID="e199634a-3078-4ca0-819d-b2714883748b" containerID="af8fd5ccf292d5766d030d9a8d95822764df87424309a41e575ea6cebbe7e453" exitCode=0
Sep 17 23:21:51.027082 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.026964    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" event={"ID":"e199634a-3078-4ca0-819d-b2714883748b","Type":"ContainerDied","Data":"af8fd5ccf292d5766d030d9a8d95822764df87424309a41e575ea6cebbe7e453"}
Sep 17 23:21:51.033547 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.033534    2582 scope.go:117] "RemoveContainer" containerID="d4f9174695686c24c933d4be7aeb504b561321a0cd942124808e0ae44dc3108d"
Sep 17 23:21:51.041518 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.041503    2582 scope.go:117] "RemoveContainer" containerID="d42bacbe144e75cd11f0b997366bb02f1384b82d9dc7d0d1d2089066c84e7c49"
Sep 17 23:21:51.052311 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.052286    2582 scope.go:117] "RemoveContainer" containerID="a6e09e52f011a61384c93513118fb646c3377145d9ee9f4327087ed70cfb9fa2"
Sep 17 23:21:51.052643 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:21:51.052623    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e09e52f011a61384c93513118fb646c3377145d9ee9f4327087ed70cfb9fa2\": container with ID starting with a6e09e52f011a61384c93513118fb646c3377145d9ee9f4327087ed70cfb9fa2 not found: ID does not exist" containerID="a6e09e52f011a61384c93513118fb646c3377145d9ee9f4327087ed70cfb9fa2"
Sep 17 23:21:51.052728 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.052666    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e09e52f011a61384c93513118fb646c3377145d9ee9f4327087ed70cfb9fa2"} err="failed to get container status \"a6e09e52f011a61384c93513118fb646c3377145d9ee9f4327087ed70cfb9fa2\": rpc error: code = NotFound desc = could not find container \"a6e09e52f011a61384c93513118fb646c3377145d9ee9f4327087ed70cfb9fa2\": container with ID starting with a6e09e52f011a61384c93513118fb646c3377145d9ee9f4327087ed70cfb9fa2 not found: ID does not exist"
Sep 17 23:21:51.052728 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.052709    2582 scope.go:117] "RemoveContainer" containerID="d4f9174695686c24c933d4be7aeb504b561321a0cd942124808e0ae44dc3108d"
Sep 17 23:21:51.053030 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:21:51.053008    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f9174695686c24c933d4be7aeb504b561321a0cd942124808e0ae44dc3108d\": container with ID starting with d4f9174695686c24c933d4be7aeb504b561321a0cd942124808e0ae44dc3108d not found: ID does not exist" containerID="d4f9174695686c24c933d4be7aeb504b561321a0cd942124808e0ae44dc3108d"
Sep 17 23:21:51.053082 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.053040    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f9174695686c24c933d4be7aeb504b561321a0cd942124808e0ae44dc3108d"} err="failed to get container status \"d4f9174695686c24c933d4be7aeb504b561321a0cd942124808e0ae44dc3108d\": rpc error: code = NotFound desc = could not find container \"d4f9174695686c24c933d4be7aeb504b561321a0cd942124808e0ae44dc3108d\": container with ID starting with d4f9174695686c24c933d4be7aeb504b561321a0cd942124808e0ae44dc3108d not found: ID does not exist"
Sep 17 23:21:51.053082 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.053062    2582 scope.go:117] "RemoveContainer" containerID="d42bacbe144e75cd11f0b997366bb02f1384b82d9dc7d0d1d2089066c84e7c49"
Sep 17 23:21:51.053398 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:21:51.053375    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42bacbe144e75cd11f0b997366bb02f1384b82d9dc7d0d1d2089066c84e7c49\": container with ID starting with d42bacbe144e75cd11f0b997366bb02f1384b82d9dc7d0d1d2089066c84e7c49 not found: ID does not exist" containerID="d42bacbe144e75cd11f0b997366bb02f1384b82d9dc7d0d1d2089066c84e7c49"
Sep 17 23:21:51.053441 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.053406    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42bacbe144e75cd11f0b997366bb02f1384b82d9dc7d0d1d2089066c84e7c49"} err="failed to get container status \"d42bacbe144e75cd11f0b997366bb02f1384b82d9dc7d0d1d2089066c84e7c49\": rpc error: code = NotFound desc = could not find container \"d42bacbe144e75cd11f0b997366bb02f1384b82d9dc7d0d1d2089066c84e7c49\": container with ID starting with d42bacbe144e75cd11f0b997366bb02f1384b82d9dc7d0d1d2089066c84e7c49 not found: ID does not exist"
Sep 17 23:21:51.075379 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.075358    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"]
Sep 17 23:21:51.082825 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.082804    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-b9d76bcd7-ck5s8"]
Sep 17 23:21:51.279986 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.279896    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"]
Sep 17 23:21:51.280373 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.280349    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="storage-initializer"
Sep 17 23:21:51.280373 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.280373    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="storage-initializer"
Sep 17 23:21:51.280525 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.280392    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kserve-container"
Sep 17 23:21:51.280525 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.280401    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kserve-container"
Sep 17 23:21:51.280525 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.280421    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kube-rbac-proxy"
Sep 17 23:21:51.280525 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.280429    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kube-rbac-proxy"
Sep 17 23:21:51.280525 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.280494    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kube-rbac-proxy"
Sep 17 23:21:51.280525 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.280504    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" containerName="kserve-container"
Sep 17 23:21:51.283991 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.283973    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.287113 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.287090    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-7b7df-kube-rbac-proxy-sar-config\""
Sep 17 23:21:51.287226 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.287134    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-7b7df-predictor-serving-cert\""
Sep 17 23:21:51.298403 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.298381    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"]
Sep 17 23:21:51.394269 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.394238    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d19dd52-dc0f-48d8-b26d-0d6018002af1" path="/var/lib/kubelet/pods/6d19dd52-dc0f-48d8-b26d-0d6018002af1/volumes"
Sep 17 23:21:51.420408 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.420375    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.420408 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.420412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.420573 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.420448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9d7s\" (UniqueName: \"kubernetes.io/projected/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kube-api-access-k9d7s\") pod \"isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.420573 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.420523    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-proxy-tls\") pod \"isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.420644 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.420612    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.457550 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.457515    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"]
Sep 17 23:21:51.461239 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.461223    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:51.464736 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.464713    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-7b7df-predictor-serving-cert\""
Sep 17 23:21:51.464893 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.464809    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-7b7df-kube-rbac-proxy-sar-config\""
Sep 17 23:21:51.477873 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.477822    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"]
Sep 17 23:21:51.521460 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.521439    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.521564 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.521470    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.521564 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.521489    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f117bceb-250e-461c-8438-60ae921845aa-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:51.521678 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.521621    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9d7s\" (UniqueName: \"kubernetes.io/projected/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kube-api-access-k9d7s\") pod \"isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.521678 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.521664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-proxy-tls\") pod \"isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.521798 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.521776    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnh2c\" (UniqueName: \"kubernetes.io/projected/f117bceb-250e-461c-8438-60ae921845aa-kube-api-access-hnh2c\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:51.521888 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.521858    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f117bceb-250e-461c-8438-60ae921845aa-cabundle-cert\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:51.521944 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.521889    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.521944 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.521904    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f117bceb-250e-461c-8438-60ae921845aa-proxy-tls\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:51.522045 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.521951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.522045 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.521969    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f117bceb-250e-461c-8438-60ae921845aa-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:51.522258 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.522237    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.522496 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.522475    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.524289 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.524262    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-proxy-tls\") pod \"isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.530860 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.530788    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9d7s\" (UniqueName: \"kubernetes.io/projected/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kube-api-access-k9d7s\") pod \"isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.594407 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.594382    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:51.623345 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.623320    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f117bceb-250e-461c-8438-60ae921845aa-cabundle-cert\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:51.623469 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.623356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f117bceb-250e-461c-8438-60ae921845aa-proxy-tls\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:51.623469 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.623379    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f117bceb-250e-461c-8438-60ae921845aa-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:51.623589 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:21:51.623500    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-serving-cert: secret "isvc-xgboost-graph-raw-7b7df-predictor-serving-cert" not found
Sep 17 23:21:51.623589 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.623522    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f117bceb-250e-461c-8438-60ae921845aa-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:51.623589 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:21:51.623574    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f117bceb-250e-461c-8438-60ae921845aa-proxy-tls podName:f117bceb-250e-461c-8438-60ae921845aa nodeName:}" failed. No retries permitted until 2026-09-17 23:21:52.1235534 +0000 UTC m=+666.810625961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f117bceb-250e-461c-8438-60ae921845aa-proxy-tls") pod "isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" (UID: "f117bceb-250e-461c-8438-60ae921845aa") : secret "isvc-xgboost-graph-raw-7b7df-predictor-serving-cert" not found
Sep 17 23:21:51.623755 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.623649    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnh2c\" (UniqueName: \"kubernetes.io/projected/f117bceb-250e-461c-8438-60ae921845aa-kube-api-access-hnh2c\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:51.624034 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.624008    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f117bceb-250e-461c-8438-60ae921845aa-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:51.624133 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.624113    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f117bceb-250e-461c-8438-60ae921845aa-cabundle-cert\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:51.624210 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.624158    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f117bceb-250e-461c-8438-60ae921845aa-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:51.634068 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.633608    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnh2c\" (UniqueName: \"kubernetes.io/projected/f117bceb-250e-461c-8438-60ae921845aa-kube-api-access-hnh2c\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:51.874875 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:51.874830    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"]
Sep 17 23:21:51.876968 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:21:51.876945    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ecb8950_e3e6_41a5_8147_e9a74a14a68c.slice/crio-8c092c0f181383dc5b6f1e2c914fac592a91c712b03f1069779335b9afe3c2e0 WatchSource:0}: Error finding container 8c092c0f181383dc5b6f1e2c914fac592a91c712b03f1069779335b9afe3c2e0: Status 404 returned error can't find the container with id 8c092c0f181383dc5b6f1e2c914fac592a91c712b03f1069779335b9afe3c2e0
Sep 17 23:21:52.033702 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:52.033665    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" event={"ID":"e199634a-3078-4ca0-819d-b2714883748b","Type":"ContainerStarted","Data":"7afff0bdb346e96cb88979e5eb8b5ed23bc0519e64f92da55f17978266077568"}
Sep 17 23:21:52.033869 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:52.033737    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" event={"ID":"e199634a-3078-4ca0-819d-b2714883748b","Type":"ContainerStarted","Data":"2f90397df0ac4568a31afd4613c6507a1cd7b54713ef5f910f0106339e10477d"}
Sep 17 23:21:52.034004 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:52.033980    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:52.035294 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:52.035271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" event={"ID":"8ecb8950-e3e6-41a5-8147-e9a74a14a68c","Type":"ContainerStarted","Data":"ad1f30634096c3c36ff50e5f5760080c0f476946e3a4dcc60029b5db78d641e8"}
Sep 17 23:21:52.035400 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:52.035317    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" event={"ID":"8ecb8950-e3e6-41a5-8147-e9a74a14a68c","Type":"ContainerStarted","Data":"8c092c0f181383dc5b6f1e2c914fac592a91c712b03f1069779335b9afe3c2e0"}
Sep 17 23:21:52.070251 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:52.070204    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" podStartSLOduration=2.460500735 podStartE2EDuration="6.070190106s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.609689371s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:52.068233095 +0000 UTC m=+666.755305684" watchObservedRunningTime="2026-09-17 23:21:52.070190106 +0000 UTC m=+666.757262718"
Sep 17 23:21:52.128090 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:52.127679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f117bceb-250e-461c-8438-60ae921845aa-proxy-tls\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:52.130323 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:52.130298    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f117bceb-250e-461c-8438-60ae921845aa-proxy-tls\") pod \"isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:52.371602 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:52.371559    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:21:52.645990 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:52.645968    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"]
Sep 17 23:21:52.648456 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:21:52.648433    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf117bceb_250e_461c_8438_60ae921845aa.slice/crio-9fc28016d48b81fa13973c1fd8d0a6e88e12167db1ec7055829eb2fddce0d234 WatchSource:0}: Error finding container 9fc28016d48b81fa13973c1fd8d0a6e88e12167db1ec7055829eb2fddce0d234: Status 404 returned error can't find the container with id 9fc28016d48b81fa13973c1fd8d0a6e88e12167db1ec7055829eb2fddce0d234
Sep 17 23:21:53.041252 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:53.041161    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" event={"ID":"f117bceb-250e-461c-8438-60ae921845aa","Type":"ContainerStarted","Data":"3438aa86523edcee3fd5357c2ac0ea72626245bf76018ff659a4f55e424e4c61"}
Sep 17 23:21:53.041252 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:53.041221    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" event={"ID":"f117bceb-250e-461c-8438-60ae921845aa","Type":"ContainerStarted","Data":"9fc28016d48b81fa13973c1fd8d0a6e88e12167db1ec7055829eb2fddce0d234"}
Sep 17 23:21:53.041720 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:53.041696    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:53.042945 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:53.042920    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 23:21:54.045456 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:54.045410    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 23:21:56.054921 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:56.054889    2582 generic.go:356] "Generic (PLEG): container finished" podID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerID="ad1f30634096c3c36ff50e5f5760080c0f476946e3a4dcc60029b5db78d641e8" exitCode=0
Sep 17 23:21:56.055378 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:56.054971    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" event={"ID":"8ecb8950-e3e6-41a5-8147-e9a74a14a68c","Type":"ContainerDied","Data":"ad1f30634096c3c36ff50e5f5760080c0f476946e3a4dcc60029b5db78d641e8"}
Sep 17 23:21:57.061345 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:57.061303    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" event={"ID":"8ecb8950-e3e6-41a5-8147-e9a74a14a68c","Type":"ContainerStarted","Data":"573ea74491913c6854709807650f06bacb9872fc282dbcb9a15c8cf9ed17ada0"}
Sep 17 23:21:57.061731 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:57.061425    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" event={"ID":"8ecb8950-e3e6-41a5-8147-e9a74a14a68c","Type":"ContainerStarted","Data":"ed988422141075e372ed9d1f6d046d93e71cae8c3689607b69c3ea47575b53f7"}
Sep 17 23:21:57.061791 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:57.061761    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:57.061889 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:57.061815    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:21:57.063010 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:57.062984    2582 generic.go:356] "Generic (PLEG): container finished" podID="f117bceb-250e-461c-8438-60ae921845aa" containerID="3438aa86523edcee3fd5357c2ac0ea72626245bf76018ff659a4f55e424e4c61" exitCode=0
Sep 17 23:21:57.063116 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:57.063033    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" event={"ID":"f117bceb-250e-461c-8438-60ae921845aa","Type":"ContainerDied","Data":"3438aa86523edcee3fd5357c2ac0ea72626245bf76018ff659a4f55e424e4c61"}
Sep 17 23:21:57.063198 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:57.063177    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 23:21:57.064273 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:57.064256    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:21:57.082579 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:57.082437    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" podStartSLOduration=2.175689015 podStartE2EDuration="6.082421228s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.906732213s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:57.081993944 +0000 UTC m=+671.769066522" watchObservedRunningTime="2026-09-17 23:21:57.082421228 +0000 UTC m=+671.769493806"
Sep 17 23:21:58.068825 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:58.068780    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 23:21:59.051514 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:59.051461    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:21:59.052500 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:21:59.052147    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 23:22:03.074659 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:03.074625    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:22:03.075342 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:03.075313    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 23:22:09.052419 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:09.052367    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 23:22:13.076269 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:13.076226    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 23:22:16.160985 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:16.160948    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" event={"ID":"f117bceb-250e-461c-8438-60ae921845aa","Type":"ContainerStarted","Data":"66d8386a4a965519649506106717c2da28c630ef720d5099db1932fa56bdce07"}
Sep 17 23:22:16.161352 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:16.161033    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" event={"ID":"f117bceb-250e-461c-8438-60ae921845aa","Type":"ContainerStarted","Data":"bffb82b60497ad955fc800e7e43f0b817ef3794dc3bd736846f66d8a6fb58c0d"}
Sep 17 23:22:16.161352 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:16.161298    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:22:16.161453 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:16.161435    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:22:16.162692 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:16.162667    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 23:22:16.181089 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:16.181046    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" podStartSLOduration=2.856994577 podStartE2EDuration="25.181033551s" totalImagesPullingTime="18.517153158s" totalInitContainerRuntime="3.806885816s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:16.180307605 +0000 UTC m=+690.867380183" watchObservedRunningTime="2026-09-17 23:22:16.181033551 +0000 UTC m=+690.868106127"
Sep 17 23:22:17.165009 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:17.164966    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 23:22:19.053052 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:19.053013    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 23:22:22.170449 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:22.170419    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:22:22.170990 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:22.170961    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 23:22:23.075900 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:23.075851    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 23:22:29.052542 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:29.052461    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 23:22:32.171386 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:32.171335    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 23:22:33.075850 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:33.075802    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 23:22:39.052710 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:39.052667    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 23:22:42.171555 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:42.171505    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 23:22:43.076160 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:43.076119    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 23:22:49.052957 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:49.052913    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 23:22:52.170975 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:52.170931    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 23:22:53.075258 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:53.075217    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 23:22:59.052786 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:22:59.052744    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 23:23:02.171821 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:02.171769    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 23:23:03.075276 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:03.075228    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 23:23:09.052985 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:09.052957    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:23:12.171403 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:12.171359    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 23:23:12.389937 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:12.389909    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:23:16.170209 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:16.170176    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"]
Sep 17 23:23:16.170695 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:16.170416    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kserve-container" containerID="cri-o://2f90397df0ac4568a31afd4613c6507a1cd7b54713ef5f910f0106339e10477d" gracePeriod=30
Sep 17 23:23:16.170695 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:16.170480    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kube-rbac-proxy" containerID="cri-o://7afff0bdb346e96cb88979e5eb8b5ed23bc0519e64f92da55f17978266077568" gracePeriod=30
Sep 17 23:23:16.402497 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:16.402459    2582 generic.go:356] "Generic (PLEG): container finished" podID="e199634a-3078-4ca0-819d-b2714883748b" containerID="7afff0bdb346e96cb88979e5eb8b5ed23bc0519e64f92da55f17978266077568" exitCode=2
Sep 17 23:23:16.402658 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:16.402532    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" event={"ID":"e199634a-3078-4ca0-819d-b2714883748b","Type":"ContainerDied","Data":"7afff0bdb346e96cb88979e5eb8b5ed23bc0519e64f92da55f17978266077568"}
Sep 17 23:23:19.045529 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:19.045490    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 17 23:23:19.052188 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:19.052155    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 17 23:23:20.420642 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.420608    2582 generic.go:356] "Generic (PLEG): container finished" podID="e199634a-3078-4ca0-819d-b2714883748b" containerID="2f90397df0ac4568a31afd4613c6507a1cd7b54713ef5f910f0106339e10477d" exitCode=0
Sep 17 23:23:20.421013 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.420651    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" event={"ID":"e199634a-3078-4ca0-819d-b2714883748b","Type":"ContainerDied","Data":"2f90397df0ac4568a31afd4613c6507a1cd7b54713ef5f910f0106339e10477d"}
Sep 17 23:23:20.510092 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.510069    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:23:20.567330 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.567298    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e199634a-3078-4ca0-819d-b2714883748b-kube-rbac-proxy-sar-config\") pod \"e199634a-3078-4ca0-819d-b2714883748b\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") "
Sep 17 23:23:20.567479 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.567355    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e199634a-3078-4ca0-819d-b2714883748b-kserve-provision-location\") pod \"e199634a-3078-4ca0-819d-b2714883748b\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") "
Sep 17 23:23:20.567479 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.567395    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78vg5\" (UniqueName: \"kubernetes.io/projected/e199634a-3078-4ca0-819d-b2714883748b-kube-api-access-78vg5\") pod \"e199634a-3078-4ca0-819d-b2714883748b\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") "
Sep 17 23:23:20.567479 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.567429    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e199634a-3078-4ca0-819d-b2714883748b-proxy-tls\") pod \"e199634a-3078-4ca0-819d-b2714883748b\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") "
Sep 17 23:23:20.567624 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.567484    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e199634a-3078-4ca0-819d-b2714883748b-cabundle-cert\") pod \"e199634a-3078-4ca0-819d-b2714883748b\" (UID: \"e199634a-3078-4ca0-819d-b2714883748b\") "
Sep 17 23:23:20.567692 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.567664    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e199634a-3078-4ca0-819d-b2714883748b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e199634a-3078-4ca0-819d-b2714883748b" (UID: "e199634a-3078-4ca0-819d-b2714883748b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:20.567735 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.567667    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e199634a-3078-4ca0-819d-b2714883748b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e199634a-3078-4ca0-819d-b2714883748b" (UID: "e199634a-3078-4ca0-819d-b2714883748b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:23:20.567909 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.567884    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e199634a-3078-4ca0-819d-b2714883748b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e199634a-3078-4ca0-819d-b2714883748b" (UID: "e199634a-3078-4ca0-819d-b2714883748b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:20.569552 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.569529    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e199634a-3078-4ca0-819d-b2714883748b-kube-api-access-78vg5" (OuterVolumeSpecName: "kube-api-access-78vg5") pod "e199634a-3078-4ca0-819d-b2714883748b" (UID: "e199634a-3078-4ca0-819d-b2714883748b"). InnerVolumeSpecName "kube-api-access-78vg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:23:20.569640 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.569553    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e199634a-3078-4ca0-819d-b2714883748b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e199634a-3078-4ca0-819d-b2714883748b" (UID: "e199634a-3078-4ca0-819d-b2714883748b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:23:20.668666 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.668636    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e199634a-3078-4ca0-819d-b2714883748b-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:23:20.668794 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.668671    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e199634a-3078-4ca0-819d-b2714883748b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:23:20.668794 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.668684    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e199634a-3078-4ca0-819d-b2714883748b-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:23:20.668794 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.668699    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78vg5\" (UniqueName: \"kubernetes.io/projected/e199634a-3078-4ca0-819d-b2714883748b-kube-api-access-78vg5\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:23:20.668794 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:20.668711    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e199634a-3078-4ca0-819d-b2714883748b-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:23:21.394288 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.394256    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"]
Sep 17 23:23:21.394630 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.394610    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kube-rbac-proxy"
Sep 17 23:23:21.394630 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.394628    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kube-rbac-proxy"
Sep 17 23:23:21.394791 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.394646    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kserve-container"
Sep 17 23:23:21.394791 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.394652    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kserve-container"
Sep 17 23:23:21.394791 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.394664    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="storage-initializer"
Sep 17 23:23:21.394791 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.394670    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="storage-initializer"
Sep 17 23:23:21.394791 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.394727    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kube-rbac-proxy"
Sep 17 23:23:21.394791 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.394736    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e199634a-3078-4ca0-819d-b2714883748b" containerName="kserve-container"
Sep 17 23:23:21.396921 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.396903    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.399441 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.399424    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 23:23:21.399626 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.399535    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 23:23:21.407223 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.407203    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"]
Sep 17 23:23:21.426843 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.426821    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"
Sep 17 23:23:21.427167 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.426859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6" event={"ID":"e199634a-3078-4ca0-819d-b2714883748b","Type":"ContainerDied","Data":"1cf11ac20e518b53f5a2b04872bfe2939fed4aa6439aaf7456e2cb53ebb960ed"}
Sep 17 23:23:21.427167 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.426930    2582 scope.go:117] "RemoveContainer" containerID="7afff0bdb346e96cb88979e5eb8b5ed23bc0519e64f92da55f17978266077568"
Sep 17 23:23:21.436061 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.436031    2582 scope.go:117] "RemoveContainer" containerID="2f90397df0ac4568a31afd4613c6507a1cd7b54713ef5f910f0106339e10477d"
Sep 17 23:23:21.443366 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.443346    2582 scope.go:117] "RemoveContainer" containerID="af8fd5ccf292d5766d030d9a8d95822764df87424309a41e575ea6cebbe7e453"
Sep 17 23:23:21.445821 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.445797    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"]
Sep 17 23:23:21.448168 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.448146    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-zw6z6"]
Sep 17 23:23:21.476052 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.476030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.476146 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.476063    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.476146 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.476084    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttdzh\" (UniqueName: \"kubernetes.io/projected/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kube-api-access-ttdzh\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.476261 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.476206    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.476261 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.476245    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.479274 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.479254    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"]
Sep 17 23:23:21.482245 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.482232    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.484784 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.484766    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 23:23:21.492751 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.492730    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"]
Sep 17 23:23:21.577467 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.577442    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98a16c6c-881e-43a0-ac41-d61b32479632-cabundle-cert\") pod \"isvc-canary-stop-predictor-596476db66-z2fhv\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.577599 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.577475    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.577599 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.577494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.577599 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.577511    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98a16c6c-881e-43a0-ac41-d61b32479632-kserve-provision-location\") pod \"isvc-canary-stop-predictor-596476db66-z2fhv\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.577777 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.577635    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mzkn\" (UniqueName: \"kubernetes.io/projected/98a16c6c-881e-43a0-ac41-d61b32479632-kube-api-access-2mzkn\") pod \"isvc-canary-stop-predictor-596476db66-z2fhv\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.577777 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.577676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.577777 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.577703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98a16c6c-881e-43a0-ac41-d61b32479632-proxy-tls\") pod \"isvc-canary-stop-predictor-596476db66-z2fhv\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.577777 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.577734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.577993 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.577792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttdzh\" (UniqueName: \"kubernetes.io/projected/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kube-api-access-ttdzh\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.577993 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.577826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98a16c6c-881e-43a0-ac41-d61b32479632-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-596476db66-z2fhv\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.577993 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.577920    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.578354 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.578337    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.578611 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.578591    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.580177 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.580161    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.588660 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.588642    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttdzh\" (UniqueName: \"kubernetes.io/projected/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kube-api-access-ttdzh\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.678991 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.678929    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98a16c6c-881e-43a0-ac41-d61b32479632-kserve-provision-location\") pod \"isvc-canary-stop-predictor-596476db66-z2fhv\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.678991 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.678964    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mzkn\" (UniqueName: \"kubernetes.io/projected/98a16c6c-881e-43a0-ac41-d61b32479632-kube-api-access-2mzkn\") pod \"isvc-canary-stop-predictor-596476db66-z2fhv\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.679147 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.678992    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98a16c6c-881e-43a0-ac41-d61b32479632-proxy-tls\") pod \"isvc-canary-stop-predictor-596476db66-z2fhv\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.679147 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.679018    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98a16c6c-881e-43a0-ac41-d61b32479632-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-596476db66-z2fhv\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.679147 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.679045    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98a16c6c-881e-43a0-ac41-d61b32479632-cabundle-cert\") pod \"isvc-canary-stop-predictor-596476db66-z2fhv\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.679319 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.679289    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98a16c6c-881e-43a0-ac41-d61b32479632-kserve-provision-location\") pod \"isvc-canary-stop-predictor-596476db66-z2fhv\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.679539 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.679521    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98a16c6c-881e-43a0-ac41-d61b32479632-cabundle-cert\") pod \"isvc-canary-stop-predictor-596476db66-z2fhv\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.679647 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.679628    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98a16c6c-881e-43a0-ac41-d61b32479632-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-596476db66-z2fhv\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.681370 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.681347    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98a16c6c-881e-43a0-ac41-d61b32479632-proxy-tls\") pod \"isvc-canary-stop-predictor-596476db66-z2fhv\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.687763 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.687740    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mzkn\" (UniqueName: \"kubernetes.io/projected/98a16c6c-881e-43a0-ac41-d61b32479632-kube-api-access-2mzkn\") pod \"isvc-canary-stop-predictor-596476db66-z2fhv\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.709172 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.709144    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:21.792266 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.792229    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:21.980016 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:21.979917    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"]
Sep 17 23:23:21.982583 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:23:21.982561    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7623cb91_093e_435e_b7ca_3bbc3fedc5d6.slice/crio-bee0c722c833e7fdf02e6d5edb02b2e7650a717c09e7027896741f3e2505cdef WatchSource:0}: Error finding container bee0c722c833e7fdf02e6d5edb02b2e7650a717c09e7027896741f3e2505cdef: Status 404 returned error can't find the container with id bee0c722c833e7fdf02e6d5edb02b2e7650a717c09e7027896741f3e2505cdef
Sep 17 23:23:22.062080 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:22.062054    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"]
Sep 17 23:23:22.064430 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:23:22.064406    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a16c6c_881e_43a0_ac41_d61b32479632.slice/crio-9205d9f52e85dc75591e3bd3e5fd926a7f5aa771fb717eeaf4a406e084d4e6fb WatchSource:0}: Error finding container 9205d9f52e85dc75591e3bd3e5fd926a7f5aa771fb717eeaf4a406e084d4e6fb: Status 404 returned error can't find the container with id 9205d9f52e85dc75591e3bd3e5fd926a7f5aa771fb717eeaf4a406e084d4e6fb
Sep 17 23:23:22.171907 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:22.171882    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:23:22.434630 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:22.434586    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" event={"ID":"98a16c6c-881e-43a0-ac41-d61b32479632","Type":"ContainerStarted","Data":"5bd2b045e4d5c0a711cca33aae9f0223ffe10f12c58327ec979b628307dcbc42"}
Sep 17 23:23:22.435116 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:22.434670    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" event={"ID":"98a16c6c-881e-43a0-ac41-d61b32479632","Type":"ContainerStarted","Data":"9205d9f52e85dc75591e3bd3e5fd926a7f5aa771fb717eeaf4a406e084d4e6fb"}
Sep 17 23:23:22.436375 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:22.436346    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" event={"ID":"7623cb91-093e-435e-b7ca-3bbc3fedc5d6","Type":"ContainerStarted","Data":"889e1449d36a7954e888781ec87f3bcac701982a24226fae03c2b510d04d4946"}
Sep 17 23:23:22.436482 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:22.436414    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" event={"ID":"7623cb91-093e-435e-b7ca-3bbc3fedc5d6","Type":"ContainerStarted","Data":"bee0c722c833e7fdf02e6d5edb02b2e7650a717c09e7027896741f3e2505cdef"}
Sep 17 23:23:23.393475 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:23.393441    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e199634a-3078-4ca0-819d-b2714883748b" path="/var/lib/kubelet/pods/e199634a-3078-4ca0-819d-b2714883748b/volumes"
Sep 17 23:23:26.452895 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:26.452859    2582 generic.go:356] "Generic (PLEG): container finished" podID="98a16c6c-881e-43a0-ac41-d61b32479632" containerID="5bd2b045e4d5c0a711cca33aae9f0223ffe10f12c58327ec979b628307dcbc42" exitCode=0
Sep 17 23:23:26.453321 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:26.452862    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" event={"ID":"98a16c6c-881e-43a0-ac41-d61b32479632","Type":"ContainerDied","Data":"5bd2b045e4d5c0a711cca33aae9f0223ffe10f12c58327ec979b628307dcbc42"}
Sep 17 23:23:26.454614 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:26.454591    2582 generic.go:356] "Generic (PLEG): container finished" podID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerID="889e1449d36a7954e888781ec87f3bcac701982a24226fae03c2b510d04d4946" exitCode=0
Sep 17 23:23:26.454709 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:26.454664    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" event={"ID":"7623cb91-093e-435e-b7ca-3bbc3fedc5d6","Type":"ContainerDied","Data":"889e1449d36a7954e888781ec87f3bcac701982a24226fae03c2b510d04d4946"}
Sep 17 23:23:27.461641 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:27.461600    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" event={"ID":"98a16c6c-881e-43a0-ac41-d61b32479632","Type":"ContainerStarted","Data":"e1caa6d4e3351b02010bb730842fdbded8acb82ca1218d3d4fa4ed288216d8e7"}
Sep 17 23:23:27.462043 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:27.461682    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" event={"ID":"98a16c6c-881e-43a0-ac41-d61b32479632","Type":"ContainerStarted","Data":"c2f836fd1f67b81f1ebc4e1a6bffed91ebf8ce203a1aa7dac50673c4d2411d28"}
Sep 17 23:23:27.462043 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:27.461918    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:27.463756 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:27.463735    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" event={"ID":"7623cb91-093e-435e-b7ca-3bbc3fedc5d6","Type":"ContainerStarted","Data":"f34eeb5bfa84789444a5cf0fac1e7962b3fb9a17174e699a6c47a86ce6bd2b46"}
Sep 17 23:23:27.463869 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:27.463780    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" event={"ID":"7623cb91-093e-435e-b7ca-3bbc3fedc5d6","Type":"ContainerStarted","Data":"067a7be024b39d60e1d3ada28f0fb421dbef96b2865ff293be8f117cfa13e58a"}
Sep 17 23:23:27.463975 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:27.463961    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:27.464013 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:27.463998    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:27.465499 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:27.465473    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 23:23:27.485101 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:27.485052    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" podStartSLOduration=2.3602374409999998 podStartE2EDuration="6.485039178s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.124801737s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:27.48415281 +0000 UTC m=+762.171225390" watchObservedRunningTime="2026-09-17 23:23:27.485039178 +0000 UTC m=+762.172111755"
Sep 17 23:23:27.505431 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:27.505395    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" podStartSLOduration=2.514554189 podStartE2EDuration="6.50537781s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.990823621s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:27.50320229 +0000 UTC m=+762.190274861" watchObservedRunningTime="2026-09-17 23:23:27.50537781 +0000 UTC m=+762.192450381"
Sep 17 23:23:28.468631 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:28.468577    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 23:23:28.469122 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:28.468733    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:28.470137 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:28.470114    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 23:23:29.472662 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:29.472621    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 23:23:33.473154 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:33.473122    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:23:33.473672 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:33.473591    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 23:23:34.476989 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:34.476959    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:23:34.477604 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:34.477579    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 23:23:41.453550 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:41.453515    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"]
Sep 17 23:23:41.454105 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:41.453813    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kserve-container" containerID="cri-o://ed988422141075e372ed9d1f6d046d93e71cae8c3689607b69c3ea47575b53f7" gracePeriod=30
Sep 17 23:23:41.454105 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:41.453893    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kube-rbac-proxy" containerID="cri-o://573ea74491913c6854709807650f06bacb9872fc282dbcb9a15c8cf9ed17ada0" gracePeriod=30
Sep 17 23:23:41.496084 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:41.496057    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"]
Sep 17 23:23:41.496318 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:41.496292    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kserve-container" containerID="cri-o://bffb82b60497ad955fc800e7e43f0b817ef3794dc3bd736846f66d8a6fb58c0d" gracePeriod=30
Sep 17 23:23:41.496400 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:41.496320    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kube-rbac-proxy" containerID="cri-o://66d8386a4a965519649506106717c2da28c630ef720d5099db1932fa56bdce07" gracePeriod=30
Sep 17 23:23:42.165888 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:42.165824    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 17 23:23:42.171205 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:42.171164    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 17 23:23:42.390169 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:42.390123    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 17 23:23:42.531310 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:42.531228    2582 generic.go:356] "Generic (PLEG): container finished" podID="f117bceb-250e-461c-8438-60ae921845aa" containerID="66d8386a4a965519649506106717c2da28c630ef720d5099db1932fa56bdce07" exitCode=2
Sep 17 23:23:42.531683 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:42.531304    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" event={"ID":"f117bceb-250e-461c-8438-60ae921845aa","Type":"ContainerDied","Data":"66d8386a4a965519649506106717c2da28c630ef720d5099db1932fa56bdce07"}
Sep 17 23:23:42.533372 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:42.533339    2582 generic.go:356] "Generic (PLEG): container finished" podID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerID="573ea74491913c6854709807650f06bacb9872fc282dbcb9a15c8cf9ed17ada0" exitCode=2
Sep 17 23:23:42.533372 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:42.533372    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" event={"ID":"8ecb8950-e3e6-41a5-8147-e9a74a14a68c","Type":"ContainerDied","Data":"573ea74491913c6854709807650f06bacb9872fc282dbcb9a15c8cf9ed17ada0"}
Sep 17 23:23:43.069799 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:43.069751    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 17 23:23:43.473981 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:43.473933    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 23:23:44.478269 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:44.478226    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 23:23:45.438417 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.438394    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:23:45.546752 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.546659    2582 generic.go:356] "Generic (PLEG): container finished" podID="f117bceb-250e-461c-8438-60ae921845aa" containerID="bffb82b60497ad955fc800e7e43f0b817ef3794dc3bd736846f66d8a6fb58c0d" exitCode=0
Sep 17 23:23:45.546752 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.546707    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"
Sep 17 23:23:45.547142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.546754    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" event={"ID":"f117bceb-250e-461c-8438-60ae921845aa","Type":"ContainerDied","Data":"bffb82b60497ad955fc800e7e43f0b817ef3794dc3bd736846f66d8a6fb58c0d"}
Sep 17 23:23:45.547142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.546818    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p" event={"ID":"f117bceb-250e-461c-8438-60ae921845aa","Type":"ContainerDied","Data":"9fc28016d48b81fa13973c1fd8d0a6e88e12167db1ec7055829eb2fddce0d234"}
Sep 17 23:23:45.547142 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.546876    2582 scope.go:117] "RemoveContainer" containerID="66d8386a4a965519649506106717c2da28c630ef720d5099db1932fa56bdce07"
Sep 17 23:23:45.554668 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.554653    2582 scope.go:117] "RemoveContainer" containerID="bffb82b60497ad955fc800e7e43f0b817ef3794dc3bd736846f66d8a6fb58c0d"
Sep 17 23:23:45.561683 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.561663    2582 scope.go:117] "RemoveContainer" containerID="3438aa86523edcee3fd5357c2ac0ea72626245bf76018ff659a4f55e424e4c61"
Sep 17 23:23:45.569579 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.569558    2582 scope.go:117] "RemoveContainer" containerID="66d8386a4a965519649506106717c2da28c630ef720d5099db1932fa56bdce07"
Sep 17 23:23:45.569937 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:23:45.569919    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d8386a4a965519649506106717c2da28c630ef720d5099db1932fa56bdce07\": container with ID starting with 66d8386a4a965519649506106717c2da28c630ef720d5099db1932fa56bdce07 not found: ID does not exist" containerID="66d8386a4a965519649506106717c2da28c630ef720d5099db1932fa56bdce07"
Sep 17 23:23:45.570024 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.569950    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d8386a4a965519649506106717c2da28c630ef720d5099db1932fa56bdce07"} err="failed to get container status \"66d8386a4a965519649506106717c2da28c630ef720d5099db1932fa56bdce07\": rpc error: code = NotFound desc = could not find container \"66d8386a4a965519649506106717c2da28c630ef720d5099db1932fa56bdce07\": container with ID starting with 66d8386a4a965519649506106717c2da28c630ef720d5099db1932fa56bdce07 not found: ID does not exist"
Sep 17 23:23:45.570024 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.569972    2582 scope.go:117] "RemoveContainer" containerID="bffb82b60497ad955fc800e7e43f0b817ef3794dc3bd736846f66d8a6fb58c0d"
Sep 17 23:23:45.570288 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:23:45.570264    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bffb82b60497ad955fc800e7e43f0b817ef3794dc3bd736846f66d8a6fb58c0d\": container with ID starting with bffb82b60497ad955fc800e7e43f0b817ef3794dc3bd736846f66d8a6fb58c0d not found: ID does not exist" containerID="bffb82b60497ad955fc800e7e43f0b817ef3794dc3bd736846f66d8a6fb58c0d"
Sep 17 23:23:45.570357 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.570298    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bffb82b60497ad955fc800e7e43f0b817ef3794dc3bd736846f66d8a6fb58c0d"} err="failed to get container status \"bffb82b60497ad955fc800e7e43f0b817ef3794dc3bd736846f66d8a6fb58c0d\": rpc error: code = NotFound desc = could not find container \"bffb82b60497ad955fc800e7e43f0b817ef3794dc3bd736846f66d8a6fb58c0d\": container with ID starting with bffb82b60497ad955fc800e7e43f0b817ef3794dc3bd736846f66d8a6fb58c0d not found: ID does not exist"
Sep 17 23:23:45.570357 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.570322    2582 scope.go:117] "RemoveContainer" containerID="3438aa86523edcee3fd5357c2ac0ea72626245bf76018ff659a4f55e424e4c61"
Sep 17 23:23:45.570606 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:23:45.570587    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3438aa86523edcee3fd5357c2ac0ea72626245bf76018ff659a4f55e424e4c61\": container with ID starting with 3438aa86523edcee3fd5357c2ac0ea72626245bf76018ff659a4f55e424e4c61 not found: ID does not exist" containerID="3438aa86523edcee3fd5357c2ac0ea72626245bf76018ff659a4f55e424e4c61"
Sep 17 23:23:45.570689 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.570615    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3438aa86523edcee3fd5357c2ac0ea72626245bf76018ff659a4f55e424e4c61"} err="failed to get container status \"3438aa86523edcee3fd5357c2ac0ea72626245bf76018ff659a4f55e424e4c61\": rpc error: code = NotFound desc = could not find container \"3438aa86523edcee3fd5357c2ac0ea72626245bf76018ff659a4f55e424e4c61\": container with ID starting with 3438aa86523edcee3fd5357c2ac0ea72626245bf76018ff659a4f55e424e4c61 not found: ID does not exist"
Sep 17 23:23:45.588241 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.588219    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f117bceb-250e-461c-8438-60ae921845aa-cabundle-cert\") pod \"f117bceb-250e-461c-8438-60ae921845aa\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") "
Sep 17 23:23:45.588321 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.588268    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnh2c\" (UniqueName: \"kubernetes.io/projected/f117bceb-250e-461c-8438-60ae921845aa-kube-api-access-hnh2c\") pod \"f117bceb-250e-461c-8438-60ae921845aa\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") "
Sep 17 23:23:45.588363 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.588324    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f117bceb-250e-461c-8438-60ae921845aa-kserve-provision-location\") pod \"f117bceb-250e-461c-8438-60ae921845aa\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") "
Sep 17 23:23:45.588514 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.588493    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f117bceb-250e-461c-8438-60ae921845aa-kube-rbac-proxy-sar-config\") pod \"f117bceb-250e-461c-8438-60ae921845aa\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") "
Sep 17 23:23:45.588582 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.588547    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f117bceb-250e-461c-8438-60ae921845aa-proxy-tls\") pod \"f117bceb-250e-461c-8438-60ae921845aa\" (UID: \"f117bceb-250e-461c-8438-60ae921845aa\") "
Sep 17 23:23:45.588582 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.588550    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f117bceb-250e-461c-8438-60ae921845aa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f117bceb-250e-461c-8438-60ae921845aa" (UID: "f117bceb-250e-461c-8438-60ae921845aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:45.588698 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.588650    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f117bceb-250e-461c-8438-60ae921845aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f117bceb-250e-461c-8438-60ae921845aa" (UID: "f117bceb-250e-461c-8438-60ae921845aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:23:45.588872 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.588825    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f117bceb-250e-461c-8438-60ae921845aa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f117bceb-250e-461c-8438-60ae921845aa" (UID: "f117bceb-250e-461c-8438-60ae921845aa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:45.588947 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.588907    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f117bceb-250e-461c-8438-60ae921845aa-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:23:45.588947 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.588927    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f117bceb-250e-461c-8438-60ae921845aa-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:23:45.590458 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.590436    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f117bceb-250e-461c-8438-60ae921845aa-kube-api-access-hnh2c" (OuterVolumeSpecName: "kube-api-access-hnh2c") pod "f117bceb-250e-461c-8438-60ae921845aa" (UID: "f117bceb-250e-461c-8438-60ae921845aa"). InnerVolumeSpecName "kube-api-access-hnh2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:23:45.590729 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.590704    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f117bceb-250e-461c-8438-60ae921845aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f117bceb-250e-461c-8438-60ae921845aa" (UID: "f117bceb-250e-461c-8438-60ae921845aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:23:45.689449 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.689417    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f117bceb-250e-461c-8438-60ae921845aa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:23:45.689449 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.689445    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f117bceb-250e-461c-8438-60ae921845aa-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:23:45.689449 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.689455    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnh2c\" (UniqueName: \"kubernetes.io/projected/f117bceb-250e-461c-8438-60ae921845aa-kube-api-access-hnh2c\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:23:45.870781 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.870748    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"]
Sep 17 23:23:45.874392 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:45.874365    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-7b7df-predictor-74f8677b5c-qlf9p"]
Sep 17 23:23:46.995793 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:46.995769    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:23:47.101173 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.101100    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kserve-provision-location\") pod \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") "
Sep 17 23:23:47.101173 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.101157    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-cabundle-cert\") pod \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") "
Sep 17 23:23:47.101173 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.101174    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-proxy-tls\") pod \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") "
Sep 17 23:23:47.101431 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.101284    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kube-rbac-proxy-sar-config\") pod \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") "
Sep 17 23:23:47.101431 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.101320    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9d7s\" (UniqueName: \"kubernetes.io/projected/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kube-api-access-k9d7s\") pod \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\" (UID: \"8ecb8950-e3e6-41a5-8147-e9a74a14a68c\") "
Sep 17 23:23:47.101531 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.101456    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ecb8950-e3e6-41a5-8147-e9a74a14a68c" (UID: "8ecb8950-e3e6-41a5-8147-e9a74a14a68c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:23:47.101576 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.101468    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ecb8950-e3e6-41a5-8147-e9a74a14a68c" (UID: "8ecb8950-e3e6-41a5-8147-e9a74a14a68c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:47.101651 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.101633    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ecb8950-e3e6-41a5-8147-e9a74a14a68c" (UID: "8ecb8950-e3e6-41a5-8147-e9a74a14a68c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:47.103410 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.103383    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kube-api-access-k9d7s" (OuterVolumeSpecName: "kube-api-access-k9d7s") pod "8ecb8950-e3e6-41a5-8147-e9a74a14a68c" (UID: "8ecb8950-e3e6-41a5-8147-e9a74a14a68c"). InnerVolumeSpecName "kube-api-access-k9d7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:23:47.103502 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.103483    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ecb8950-e3e6-41a5-8147-e9a74a14a68c" (UID: "8ecb8950-e3e6-41a5-8147-e9a74a14a68c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:23:47.202391 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.202362    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:23:47.202518 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.202388    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9d7s\" (UniqueName: \"kubernetes.io/projected/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kube-api-access-k9d7s\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:23:47.202518 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.202410    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:23:47.202518 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.202423    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:23:47.202518 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.202435    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ecb8950-e3e6-41a5-8147-e9a74a14a68c-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:23:47.394011 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.393977    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f117bceb-250e-461c-8438-60ae921845aa" path="/var/lib/kubelet/pods/f117bceb-250e-461c-8438-60ae921845aa/volumes"
Sep 17 23:23:47.558751 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.558715    2582 generic.go:356] "Generic (PLEG): container finished" podID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerID="ed988422141075e372ed9d1f6d046d93e71cae8c3689607b69c3ea47575b53f7" exitCode=0
Sep 17 23:23:47.558933 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.558765    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" event={"ID":"8ecb8950-e3e6-41a5-8147-e9a74a14a68c","Type":"ContainerDied","Data":"ed988422141075e372ed9d1f6d046d93e71cae8c3689607b69c3ea47575b53f7"}
Sep 17 23:23:47.558933 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.558793    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"
Sep 17 23:23:47.558933 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.558873    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr" event={"ID":"8ecb8950-e3e6-41a5-8147-e9a74a14a68c","Type":"ContainerDied","Data":"8c092c0f181383dc5b6f1e2c914fac592a91c712b03f1069779335b9afe3c2e0"}
Sep 17 23:23:47.558933 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.558911    2582 scope.go:117] "RemoveContainer" containerID="573ea74491913c6854709807650f06bacb9872fc282dbcb9a15c8cf9ed17ada0"
Sep 17 23:23:47.566585 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.566563    2582 scope.go:117] "RemoveContainer" containerID="ed988422141075e372ed9d1f6d046d93e71cae8c3689607b69c3ea47575b53f7"
Sep 17 23:23:47.575245 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.575228    2582 scope.go:117] "RemoveContainer" containerID="ad1f30634096c3c36ff50e5f5760080c0f476946e3a4dcc60029b5db78d641e8"
Sep 17 23:23:47.582445 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.582420    2582 scope.go:117] "RemoveContainer" containerID="573ea74491913c6854709807650f06bacb9872fc282dbcb9a15c8cf9ed17ada0"
Sep 17 23:23:47.582681 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:23:47.582659    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573ea74491913c6854709807650f06bacb9872fc282dbcb9a15c8cf9ed17ada0\": container with ID starting with 573ea74491913c6854709807650f06bacb9872fc282dbcb9a15c8cf9ed17ada0 not found: ID does not exist" containerID="573ea74491913c6854709807650f06bacb9872fc282dbcb9a15c8cf9ed17ada0"
Sep 17 23:23:47.582759 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.582692    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573ea74491913c6854709807650f06bacb9872fc282dbcb9a15c8cf9ed17ada0"} err="failed to get container status \"573ea74491913c6854709807650f06bacb9872fc282dbcb9a15c8cf9ed17ada0\": rpc error: code = NotFound desc = could not find container \"573ea74491913c6854709807650f06bacb9872fc282dbcb9a15c8cf9ed17ada0\": container with ID starting with 573ea74491913c6854709807650f06bacb9872fc282dbcb9a15c8cf9ed17ada0 not found: ID does not exist"
Sep 17 23:23:47.582759 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.582715    2582 scope.go:117] "RemoveContainer" containerID="ed988422141075e372ed9d1f6d046d93e71cae8c3689607b69c3ea47575b53f7"
Sep 17 23:23:47.583008 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:23:47.582991    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed988422141075e372ed9d1f6d046d93e71cae8c3689607b69c3ea47575b53f7\": container with ID starting with ed988422141075e372ed9d1f6d046d93e71cae8c3689607b69c3ea47575b53f7 not found: ID does not exist" containerID="ed988422141075e372ed9d1f6d046d93e71cae8c3689607b69c3ea47575b53f7"
Sep 17 23:23:47.583053 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.583014    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed988422141075e372ed9d1f6d046d93e71cae8c3689607b69c3ea47575b53f7"} err="failed to get container status \"ed988422141075e372ed9d1f6d046d93e71cae8c3689607b69c3ea47575b53f7\": rpc error: code = NotFound desc = could not find container \"ed988422141075e372ed9d1f6d046d93e71cae8c3689607b69c3ea47575b53f7\": container with ID starting with ed988422141075e372ed9d1f6d046d93e71cae8c3689607b69c3ea47575b53f7 not found: ID does not exist"
Sep 17 23:23:47.583053 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.583030    2582 scope.go:117] "RemoveContainer" containerID="ad1f30634096c3c36ff50e5f5760080c0f476946e3a4dcc60029b5db78d641e8"
Sep 17 23:23:47.583294 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:23:47.583278    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad1f30634096c3c36ff50e5f5760080c0f476946e3a4dcc60029b5db78d641e8\": container with ID starting with ad1f30634096c3c36ff50e5f5760080c0f476946e3a4dcc60029b5db78d641e8 not found: ID does not exist" containerID="ad1f30634096c3c36ff50e5f5760080c0f476946e3a4dcc60029b5db78d641e8"
Sep 17 23:23:47.583352 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.583302    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1f30634096c3c36ff50e5f5760080c0f476946e3a4dcc60029b5db78d641e8"} err="failed to get container status \"ad1f30634096c3c36ff50e5f5760080c0f476946e3a4dcc60029b5db78d641e8\": rpc error: code = NotFound desc = could not find container \"ad1f30634096c3c36ff50e5f5760080c0f476946e3a4dcc60029b5db78d641e8\": container with ID starting with ad1f30634096c3c36ff50e5f5760080c0f476946e3a4dcc60029b5db78d641e8 not found: ID does not exist"
Sep 17 23:23:47.593048 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.593028    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"]
Sep 17 23:23:47.599871 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:47.599851    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-7b7df-predictor-6f4bcc5bc9-tkjzr"]
Sep 17 23:23:49.393368 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:49.393336    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" path="/var/lib/kubelet/pods/8ecb8950-e3e6-41a5-8147-e9a74a14a68c/volumes"
Sep 17 23:23:53.473558 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:53.473517    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 23:23:54.477675 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:23:54.477636    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 23:24:03.473760 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:03.473654    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 23:24:04.477882 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:04.477822    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 23:24:13.474079 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:13.474039    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 23:24:14.478148 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:14.478111    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 23:24:23.473795 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:23.473742    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 23:24:24.477578 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:24.477531    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 23:24:33.473711 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:33.473658    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 23:24:34.477568 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:34.477528    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 17 23:24:43.473975 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:43.473948    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:24:43.604936 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:43.604898    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"]
Sep 17 23:24:43.605188 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:43.605163    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kserve-container" containerID="cri-o://c2f836fd1f67b81f1ebc4e1a6bffed91ebf8ce203a1aa7dac50673c4d2411d28" gracePeriod=30
Sep 17 23:24:43.605260 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:43.605209    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kube-rbac-proxy" containerID="cri-o://e1caa6d4e3351b02010bb730842fdbded8acb82ca1218d3d4fa4ed288216d8e7" gracePeriod=30
Sep 17 23:24:43.606019 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:43.606000    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:24:43.786886 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:43.786761    2582 generic.go:356] "Generic (PLEG): container finished" podID="98a16c6c-881e-43a0-ac41-d61b32479632" containerID="e1caa6d4e3351b02010bb730842fdbded8acb82ca1218d3d4fa4ed288216d8e7" exitCode=2
Sep 17 23:24:43.786886 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:43.786828    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" event={"ID":"98a16c6c-881e-43a0-ac41-d61b32479632","Type":"ContainerDied","Data":"e1caa6d4e3351b02010bb730842fdbded8acb82ca1218d3d4fa4ed288216d8e7"}
Sep 17 23:24:44.473479 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:44.473433    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 17 23:24:48.450784 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.450763    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:24:48.584134 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.584064    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mzkn\" (UniqueName: \"kubernetes.io/projected/98a16c6c-881e-43a0-ac41-d61b32479632-kube-api-access-2mzkn\") pod \"98a16c6c-881e-43a0-ac41-d61b32479632\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") "
Sep 17 23:24:48.584134 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.584101    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98a16c6c-881e-43a0-ac41-d61b32479632-cabundle-cert\") pod \"98a16c6c-881e-43a0-ac41-d61b32479632\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") "
Sep 17 23:24:48.584307 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.584138    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98a16c6c-881e-43a0-ac41-d61b32479632-kserve-provision-location\") pod \"98a16c6c-881e-43a0-ac41-d61b32479632\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") "
Sep 17 23:24:48.584307 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.584169    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98a16c6c-881e-43a0-ac41-d61b32479632-proxy-tls\") pod \"98a16c6c-881e-43a0-ac41-d61b32479632\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") "
Sep 17 23:24:48.584307 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.584193    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98a16c6c-881e-43a0-ac41-d61b32479632-kube-rbac-proxy-sar-config\") pod \"98a16c6c-881e-43a0-ac41-d61b32479632\" (UID: \"98a16c6c-881e-43a0-ac41-d61b32479632\") "
Sep 17 23:24:48.584531 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.584443    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a16c6c-881e-43a0-ac41-d61b32479632-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "98a16c6c-881e-43a0-ac41-d61b32479632" (UID: "98a16c6c-881e-43a0-ac41-d61b32479632"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:24:48.584531 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.584469    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98a16c6c-881e-43a0-ac41-d61b32479632-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98a16c6c-881e-43a0-ac41-d61b32479632" (UID: "98a16c6c-881e-43a0-ac41-d61b32479632"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:24:48.584696 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.584580    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a16c6c-881e-43a0-ac41-d61b32479632-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "98a16c6c-881e-43a0-ac41-d61b32479632" (UID: "98a16c6c-881e-43a0-ac41-d61b32479632"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:24:48.586384 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.586362    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a16c6c-881e-43a0-ac41-d61b32479632-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98a16c6c-881e-43a0-ac41-d61b32479632" (UID: "98a16c6c-881e-43a0-ac41-d61b32479632"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:24:48.586468 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.586416    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a16c6c-881e-43a0-ac41-d61b32479632-kube-api-access-2mzkn" (OuterVolumeSpecName: "kube-api-access-2mzkn") pod "98a16c6c-881e-43a0-ac41-d61b32479632" (UID: "98a16c6c-881e-43a0-ac41-d61b32479632"). InnerVolumeSpecName "kube-api-access-2mzkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:24:48.685443 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.685417    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mzkn\" (UniqueName: \"kubernetes.io/projected/98a16c6c-881e-43a0-ac41-d61b32479632-kube-api-access-2mzkn\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:24:48.685443 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.685439    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98a16c6c-881e-43a0-ac41-d61b32479632-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:24:48.685582 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.685448    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98a16c6c-881e-43a0-ac41-d61b32479632-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:24:48.685582 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.685457    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98a16c6c-881e-43a0-ac41-d61b32479632-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:24:48.685582 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.685468    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98a16c6c-881e-43a0-ac41-d61b32479632-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:24:48.808182 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.808145    2582 generic.go:356] "Generic (PLEG): container finished" podID="98a16c6c-881e-43a0-ac41-d61b32479632" containerID="c2f836fd1f67b81f1ebc4e1a6bffed91ebf8ce203a1aa7dac50673c4d2411d28" exitCode=0
Sep 17 23:24:48.808354 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.808188    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" event={"ID":"98a16c6c-881e-43a0-ac41-d61b32479632","Type":"ContainerDied","Data":"c2f836fd1f67b81f1ebc4e1a6bffed91ebf8ce203a1aa7dac50673c4d2411d28"}
Sep 17 23:24:48.808354 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.808224    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"
Sep 17 23:24:48.808354 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.808274    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv" event={"ID":"98a16c6c-881e-43a0-ac41-d61b32479632","Type":"ContainerDied","Data":"9205d9f52e85dc75591e3bd3e5fd926a7f5aa771fb717eeaf4a406e084d4e6fb"}
Sep 17 23:24:48.808354 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.808296    2582 scope.go:117] "RemoveContainer" containerID="e1caa6d4e3351b02010bb730842fdbded8acb82ca1218d3d4fa4ed288216d8e7"
Sep 17 23:24:48.817263 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.817243    2582 scope.go:117] "RemoveContainer" containerID="c2f836fd1f67b81f1ebc4e1a6bffed91ebf8ce203a1aa7dac50673c4d2411d28"
Sep 17 23:24:48.824501 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.824485    2582 scope.go:117] "RemoveContainer" containerID="5bd2b045e4d5c0a711cca33aae9f0223ffe10f12c58327ec979b628307dcbc42"
Sep 17 23:24:48.831448 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.831432    2582 scope.go:117] "RemoveContainer" containerID="e1caa6d4e3351b02010bb730842fdbded8acb82ca1218d3d4fa4ed288216d8e7"
Sep 17 23:24:48.831706 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:24:48.831688    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1caa6d4e3351b02010bb730842fdbded8acb82ca1218d3d4fa4ed288216d8e7\": container with ID starting with e1caa6d4e3351b02010bb730842fdbded8acb82ca1218d3d4fa4ed288216d8e7 not found: ID does not exist" containerID="e1caa6d4e3351b02010bb730842fdbded8acb82ca1218d3d4fa4ed288216d8e7"
Sep 17 23:24:48.831749 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.831714    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1caa6d4e3351b02010bb730842fdbded8acb82ca1218d3d4fa4ed288216d8e7"} err="failed to get container status \"e1caa6d4e3351b02010bb730842fdbded8acb82ca1218d3d4fa4ed288216d8e7\": rpc error: code = NotFound desc = could not find container \"e1caa6d4e3351b02010bb730842fdbded8acb82ca1218d3d4fa4ed288216d8e7\": container with ID starting with e1caa6d4e3351b02010bb730842fdbded8acb82ca1218d3d4fa4ed288216d8e7 not found: ID does not exist"
Sep 17 23:24:48.831749 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.831728    2582 scope.go:117] "RemoveContainer" containerID="c2f836fd1f67b81f1ebc4e1a6bffed91ebf8ce203a1aa7dac50673c4d2411d28"
Sep 17 23:24:48.832010 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:24:48.831991    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f836fd1f67b81f1ebc4e1a6bffed91ebf8ce203a1aa7dac50673c4d2411d28\": container with ID starting with c2f836fd1f67b81f1ebc4e1a6bffed91ebf8ce203a1aa7dac50673c4d2411d28 not found: ID does not exist" containerID="c2f836fd1f67b81f1ebc4e1a6bffed91ebf8ce203a1aa7dac50673c4d2411d28"
Sep 17 23:24:48.832066 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.832019    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f836fd1f67b81f1ebc4e1a6bffed91ebf8ce203a1aa7dac50673c4d2411d28"} err="failed to get container status \"c2f836fd1f67b81f1ebc4e1a6bffed91ebf8ce203a1aa7dac50673c4d2411d28\": rpc error: code = NotFound desc = could not find container \"c2f836fd1f67b81f1ebc4e1a6bffed91ebf8ce203a1aa7dac50673c4d2411d28\": container with ID starting with c2f836fd1f67b81f1ebc4e1a6bffed91ebf8ce203a1aa7dac50673c4d2411d28 not found: ID does not exist"
Sep 17 23:24:48.832066 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.832037    2582 scope.go:117] "RemoveContainer" containerID="5bd2b045e4d5c0a711cca33aae9f0223ffe10f12c58327ec979b628307dcbc42"
Sep 17 23:24:48.832251 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:24:48.832236    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd2b045e4d5c0a711cca33aae9f0223ffe10f12c58327ec979b628307dcbc42\": container with ID starting with 5bd2b045e4d5c0a711cca33aae9f0223ffe10f12c58327ec979b628307dcbc42 not found: ID does not exist" containerID="5bd2b045e4d5c0a711cca33aae9f0223ffe10f12c58327ec979b628307dcbc42"
Sep 17 23:24:48.832290 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.832255    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd2b045e4d5c0a711cca33aae9f0223ffe10f12c58327ec979b628307dcbc42"} err="failed to get container status \"5bd2b045e4d5c0a711cca33aae9f0223ffe10f12c58327ec979b628307dcbc42\": rpc error: code = NotFound desc = could not find container \"5bd2b045e4d5c0a711cca33aae9f0223ffe10f12c58327ec979b628307dcbc42\": container with ID starting with 5bd2b045e4d5c0a711cca33aae9f0223ffe10f12c58327ec979b628307dcbc42 not found: ID does not exist"
Sep 17 23:24:48.837371 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.837326    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"]
Sep 17 23:24:48.843455 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:48.843436    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-z2fhv"]
Sep 17 23:24:49.393970 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:49.393941    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" path="/var/lib/kubelet/pods/98a16c6c-881e-43a0-ac41-d61b32479632/volumes"
Sep 17 23:24:51.387227 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:51.387198    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"]
Sep 17 23:24:51.387602 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:51.387436    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kserve-container" containerID="cri-o://067a7be024b39d60e1d3ada28f0fb421dbef96b2865ff293be8f117cfa13e58a" gracePeriod=30
Sep 17 23:24:51.387602 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:51.387461    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kube-rbac-proxy" containerID="cri-o://f34eeb5bfa84789444a5cf0fac1e7962b3fb9a17174e699a6c47a86ce6bd2b46" gracePeriod=30
Sep 17 23:24:51.823658 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:51.823555    2582 generic.go:356] "Generic (PLEG): container finished" podID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerID="f34eeb5bfa84789444a5cf0fac1e7962b3fb9a17174e699a6c47a86ce6bd2b46" exitCode=2
Sep 17 23:24:51.823658 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:51.823609    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" event={"ID":"7623cb91-093e-435e-b7ca-3bbc3fedc5d6","Type":"ContainerDied","Data":"f34eeb5bfa84789444a5cf0fac1e7962b3fb9a17174e699a6c47a86ce6bd2b46"}
Sep 17 23:24:53.469078 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:53.469017    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 17 23:24:53.474570 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:53.474537    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 17 23:24:56.529385 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.529361    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:24:56.650471 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.650452    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kube-rbac-proxy-sar-config\") pod \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") "
Sep 17 23:24:56.650598 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.650483    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-cabundle-cert\") pod \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") "
Sep 17 23:24:56.650598 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.650520    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttdzh\" (UniqueName: \"kubernetes.io/projected/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kube-api-access-ttdzh\") pod \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") "
Sep 17 23:24:56.650598 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.650569    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kserve-provision-location\") pod \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") "
Sep 17 23:24:56.650598 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.650588    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-proxy-tls\") pod \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\" (UID: \"7623cb91-093e-435e-b7ca-3bbc3fedc5d6\") "
Sep 17 23:24:56.650892 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.650867    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7623cb91-093e-435e-b7ca-3bbc3fedc5d6" (UID: "7623cb91-093e-435e-b7ca-3bbc3fedc5d6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:24:56.650892 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.650878    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7623cb91-093e-435e-b7ca-3bbc3fedc5d6" (UID: "7623cb91-093e-435e-b7ca-3bbc3fedc5d6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:24:56.651013 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.650921    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7623cb91-093e-435e-b7ca-3bbc3fedc5d6" (UID: "7623cb91-093e-435e-b7ca-3bbc3fedc5d6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:24:56.652880 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.652859    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7623cb91-093e-435e-b7ca-3bbc3fedc5d6" (UID: "7623cb91-093e-435e-b7ca-3bbc3fedc5d6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:24:56.652977 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.652958    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kube-api-access-ttdzh" (OuterVolumeSpecName: "kube-api-access-ttdzh") pod "7623cb91-093e-435e-b7ca-3bbc3fedc5d6" (UID: "7623cb91-093e-435e-b7ca-3bbc3fedc5d6"). InnerVolumeSpecName "kube-api-access-ttdzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:24:56.751263 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.751237    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:24:56.751263 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.751263    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:24:56.751415 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.751278    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:24:56.751415 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.751289    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:24:56.751415 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.751300    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttdzh\" (UniqueName: \"kubernetes.io/projected/7623cb91-093e-435e-b7ca-3bbc3fedc5d6-kube-api-access-ttdzh\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:24:56.844408 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.844380    2582 generic.go:356] "Generic (PLEG): container finished" podID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerID="067a7be024b39d60e1d3ada28f0fb421dbef96b2865ff293be8f117cfa13e58a" exitCode=0
Sep 17 23:24:56.844532 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.844441    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"
Sep 17 23:24:56.844532 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.844468    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" event={"ID":"7623cb91-093e-435e-b7ca-3bbc3fedc5d6","Type":"ContainerDied","Data":"067a7be024b39d60e1d3ada28f0fb421dbef96b2865ff293be8f117cfa13e58a"}
Sep 17 23:24:56.844532 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.844531    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr" event={"ID":"7623cb91-093e-435e-b7ca-3bbc3fedc5d6","Type":"ContainerDied","Data":"bee0c722c833e7fdf02e6d5edb02b2e7650a717c09e7027896741f3e2505cdef"}
Sep 17 23:24:56.844697 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.844557    2582 scope.go:117] "RemoveContainer" containerID="f34eeb5bfa84789444a5cf0fac1e7962b3fb9a17174e699a6c47a86ce6bd2b46"
Sep 17 23:24:56.853334 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.853318    2582 scope.go:117] "RemoveContainer" containerID="067a7be024b39d60e1d3ada28f0fb421dbef96b2865ff293be8f117cfa13e58a"
Sep 17 23:24:56.860663 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.860649    2582 scope.go:117] "RemoveContainer" containerID="889e1449d36a7954e888781ec87f3bcac701982a24226fae03c2b510d04d4946"
Sep 17 23:24:56.866386 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.866363    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"]
Sep 17 23:24:56.867767 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.867755    2582 scope.go:117] "RemoveContainer" containerID="f34eeb5bfa84789444a5cf0fac1e7962b3fb9a17174e699a6c47a86ce6bd2b46"
Sep 17 23:24:56.868031 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:24:56.868016    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f34eeb5bfa84789444a5cf0fac1e7962b3fb9a17174e699a6c47a86ce6bd2b46\": container with ID starting with f34eeb5bfa84789444a5cf0fac1e7962b3fb9a17174e699a6c47a86ce6bd2b46 not found: ID does not exist" containerID="f34eeb5bfa84789444a5cf0fac1e7962b3fb9a17174e699a6c47a86ce6bd2b46"
Sep 17 23:24:56.868071 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.868037    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34eeb5bfa84789444a5cf0fac1e7962b3fb9a17174e699a6c47a86ce6bd2b46"} err="failed to get container status \"f34eeb5bfa84789444a5cf0fac1e7962b3fb9a17174e699a6c47a86ce6bd2b46\": rpc error: code = NotFound desc = could not find container \"f34eeb5bfa84789444a5cf0fac1e7962b3fb9a17174e699a6c47a86ce6bd2b46\": container with ID starting with f34eeb5bfa84789444a5cf0fac1e7962b3fb9a17174e699a6c47a86ce6bd2b46 not found: ID does not exist"
Sep 17 23:24:56.868071 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.868058    2582 scope.go:117] "RemoveContainer" containerID="067a7be024b39d60e1d3ada28f0fb421dbef96b2865ff293be8f117cfa13e58a"
Sep 17 23:24:56.868295 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:24:56.868277    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"067a7be024b39d60e1d3ada28f0fb421dbef96b2865ff293be8f117cfa13e58a\": container with ID starting with 067a7be024b39d60e1d3ada28f0fb421dbef96b2865ff293be8f117cfa13e58a not found: ID does not exist" containerID="067a7be024b39d60e1d3ada28f0fb421dbef96b2865ff293be8f117cfa13e58a"
Sep 17 23:24:56.868343 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.868300    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"067a7be024b39d60e1d3ada28f0fb421dbef96b2865ff293be8f117cfa13e58a"} err="failed to get container status \"067a7be024b39d60e1d3ada28f0fb421dbef96b2865ff293be8f117cfa13e58a\": rpc error: code = NotFound desc = could not find container \"067a7be024b39d60e1d3ada28f0fb421dbef96b2865ff293be8f117cfa13e58a\": container with ID starting with 067a7be024b39d60e1d3ada28f0fb421dbef96b2865ff293be8f117cfa13e58a not found: ID does not exist"
Sep 17 23:24:56.868343 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.868316    2582 scope.go:117] "RemoveContainer" containerID="889e1449d36a7954e888781ec87f3bcac701982a24226fae03c2b510d04d4946"
Sep 17 23:24:56.868569 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:24:56.868551    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889e1449d36a7954e888781ec87f3bcac701982a24226fae03c2b510d04d4946\": container with ID starting with 889e1449d36a7954e888781ec87f3bcac701982a24226fae03c2b510d04d4946 not found: ID does not exist" containerID="889e1449d36a7954e888781ec87f3bcac701982a24226fae03c2b510d04d4946"
Sep 17 23:24:56.868635 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.868578    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889e1449d36a7954e888781ec87f3bcac701982a24226fae03c2b510d04d4946"} err="failed to get container status \"889e1449d36a7954e888781ec87f3bcac701982a24226fae03c2b510d04d4946\": rpc error: code = NotFound desc = could not find container \"889e1449d36a7954e888781ec87f3bcac701982a24226fae03c2b510d04d4946\": container with ID starting with 889e1449d36a7954e888781ec87f3bcac701982a24226fae03c2b510d04d4946 not found: ID does not exist"
Sep 17 23:24:56.871673 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:56.871654    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-qc4vr"]
Sep 17 23:24:57.393321 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:24:57.393290    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" path="/var/lib/kubelet/pods/7623cb91-093e-435e-b7ca-3bbc3fedc5d6/volumes"
Sep 17 23:25:01.661951 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.661920    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"]
Sep 17 23:25:01.662310 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662289    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="storage-initializer"
Sep 17 23:25:01.662310 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662303    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="storage-initializer"
Sep 17 23:25:01.662385 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662315    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="storage-initializer"
Sep 17 23:25:01.662385 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662324    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="storage-initializer"
Sep 17 23:25:01.662385 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662341    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="storage-initializer"
Sep 17 23:25:01.662385 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662350    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="storage-initializer"
Sep 17 23:25:01.662385 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662358    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kube-rbac-proxy"
Sep 17 23:25:01.662385 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662366    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kube-rbac-proxy"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662422    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kserve-container"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662430    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kserve-container"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662441    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="storage-initializer"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662449    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="storage-initializer"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662458    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kube-rbac-proxy"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662466    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kube-rbac-proxy"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662476    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kserve-container"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662485    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kserve-container"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662495    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kube-rbac-proxy"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662502    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kube-rbac-proxy"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662515    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kserve-container"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662522    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kserve-container"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662531    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kube-rbac-proxy"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662539    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kube-rbac-proxy"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662551    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kserve-container"
Sep 17 23:25:01.662563 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662558    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kserve-container"
Sep 17 23:25:01.663096 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662622    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kserve-container"
Sep 17 23:25:01.663096 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662632    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kserve-container"
Sep 17 23:25:01.663096 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662641    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kube-rbac-proxy"
Sep 17 23:25:01.663096 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662649    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecb8950-e3e6-41a5-8147-e9a74a14a68c" containerName="kserve-container"
Sep 17 23:25:01.663096 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662656    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f117bceb-250e-461c-8438-60ae921845aa" containerName="kube-rbac-proxy"
Sep 17 23:25:01.663096 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662664    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a16c6c-881e-43a0-ac41-d61b32479632" containerName="kube-rbac-proxy"
Sep 17 23:25:01.663096 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662672    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kserve-container"
Sep 17 23:25:01.663096 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.662682    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7623cb91-093e-435e-b7ca-3bbc3fedc5d6" containerName="kube-rbac-proxy"
Sep 17 23:25:01.667677 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.667651    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.670195 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.670170    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:25:01.670344 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.670210    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-67d4dc-kube-rbac-proxy-sar-config\""
Sep 17 23:25:01.670344 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.670245    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:25:01.670474 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.670431    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-67d4dc-predictor-serving-cert\""
Sep 17 23:25:01.670533 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.670518    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f2cbg\""
Sep 17 23:25:01.671184 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.671165    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:25:01.671267 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.671201    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:25:01.675740 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.675719    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"]
Sep 17 23:25:01.790510 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.790474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-cabundle-cert\") pod \"isvc-primary-67d4dc-predictor-774f887c4b-h476t\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") " pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.790663 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.790513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf5p8\" (UniqueName: \"kubernetes.io/projected/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kube-api-access-bf5p8\") pod \"isvc-primary-67d4dc-predictor-774f887c4b-h476t\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") " pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.790663 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.790548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kube-rbac-proxy-sar-config\") pod \"isvc-primary-67d4dc-predictor-774f887c4b-h476t\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") " pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.790663 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.790600    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-proxy-tls\") pod \"isvc-primary-67d4dc-predictor-774f887c4b-h476t\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") " pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.790663 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.790632    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kserve-provision-location\") pod \"isvc-primary-67d4dc-predictor-774f887c4b-h476t\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") " pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.891480 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.891441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-cabundle-cert\") pod \"isvc-primary-67d4dc-predictor-774f887c4b-h476t\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") " pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.891480 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.891483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf5p8\" (UniqueName: \"kubernetes.io/projected/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kube-api-access-bf5p8\") pod \"isvc-primary-67d4dc-predictor-774f887c4b-h476t\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") " pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.891731 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.891501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kube-rbac-proxy-sar-config\") pod \"isvc-primary-67d4dc-predictor-774f887c4b-h476t\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") " pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.891731 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.891563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-proxy-tls\") pod \"isvc-primary-67d4dc-predictor-774f887c4b-h476t\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") " pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.891731 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.891598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kserve-provision-location\") pod \"isvc-primary-67d4dc-predictor-774f887c4b-h476t\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") " pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.892010 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.891988    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kserve-provision-location\") pod \"isvc-primary-67d4dc-predictor-774f887c4b-h476t\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") " pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.892152 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.892136    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kube-rbac-proxy-sar-config\") pod \"isvc-primary-67d4dc-predictor-774f887c4b-h476t\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") " pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.892200 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.892179    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-cabundle-cert\") pod \"isvc-primary-67d4dc-predictor-774f887c4b-h476t\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") " pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.894233 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.894209    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-proxy-tls\") pod \"isvc-primary-67d4dc-predictor-774f887c4b-h476t\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") " pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.900329 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.900305    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf5p8\" (UniqueName: \"kubernetes.io/projected/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kube-api-access-bf5p8\") pod \"isvc-primary-67d4dc-predictor-774f887c4b-h476t\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") " pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:01.978233 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:01.978145    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:02.249935 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:02.249908    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"]
Sep 17 23:25:02.252461 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:25:02.252418    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35541dae_4ee3_4b82_8b5d_fbe6d5e1dbaa.slice/crio-62b3ecfc5d0fe927b9f9f575e949c9b2041630371cb1e4dbc83837d789cddf82 WatchSource:0}: Error finding container 62b3ecfc5d0fe927b9f9f575e949c9b2041630371cb1e4dbc83837d789cddf82: Status 404 returned error can't find the container with id 62b3ecfc5d0fe927b9f9f575e949c9b2041630371cb1e4dbc83837d789cddf82
Sep 17 23:25:02.868562 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:02.868522    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" event={"ID":"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa","Type":"ContainerStarted","Data":"6f2b18e28f40ef87c570b90c1fcad4b2f19f873bbf864c45cdb5cfdb69b7ca77"}
Sep 17 23:25:02.868937 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:02.868612    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" event={"ID":"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa","Type":"ContainerStarted","Data":"62b3ecfc5d0fe927b9f9f575e949c9b2041630371cb1e4dbc83837d789cddf82"}
Sep 17 23:25:06.886743 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:06.886709    2582 generic.go:356] "Generic (PLEG): container finished" podID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerID="6f2b18e28f40ef87c570b90c1fcad4b2f19f873bbf864c45cdb5cfdb69b7ca77" exitCode=0
Sep 17 23:25:06.887133 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:06.886783    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" event={"ID":"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa","Type":"ContainerDied","Data":"6f2b18e28f40ef87c570b90c1fcad4b2f19f873bbf864c45cdb5cfdb69b7ca77"}
Sep 17 23:25:07.892791 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:07.892756    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" event={"ID":"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa","Type":"ContainerStarted","Data":"99463aa085e18b122d42f8f721c0718732620a116ef0eab4f9ca945c4294a6bb"}
Sep 17 23:25:07.893234 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:07.892820    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" event={"ID":"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa","Type":"ContainerStarted","Data":"d0c814b33f40cd930893d5c2e300fff5884fc7db7d76946934fa375ee4581f55"}
Sep 17 23:25:07.893234 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:07.893120    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:07.893306 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:07.893251    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:07.894567 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:07.894538    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 23:25:07.912824 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:07.912780    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" podStartSLOduration=2.994381625 podStartE2EDuration="6.912767619s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.918385994s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:07.911747984 +0000 UTC m=+862.598820587" watchObservedRunningTime="2026-09-17 23:25:07.912767619 +0000 UTC m=+862.599840196"
Sep 17 23:25:08.897250 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:08.897211    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 23:25:13.902174 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:13.902143    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:25:13.902759 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:13.902732    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 23:25:23.902722 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:23.902680    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 23:25:31.813490 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:31.813387    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"]
Sep 17 23:25:31.817145 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:31.817121    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:25:31.819751 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:31.819730    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-0a0da-predictor-serving-cert\""
Sep 17 23:25:31.819751 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:31.819741    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-0a0da-kube-rbac-proxy-sar-config\""
Sep 17 23:25:31.826998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:31.826965    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"]
Sep 17 23:25:31.833765 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:31.833736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l857v\" (UniqueName: \"kubernetes.io/projected/d8f62f64-a4ef-4506-8d68-5803bed81380-kube-api-access-l857v\") pod \"message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26\" (UID: \"d8f62f64-a4ef-4506-8d68-5803bed81380\") " pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:25:31.833908 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:31.833796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8f62f64-a4ef-4506-8d68-5803bed81380-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26\" (UID: \"d8f62f64-a4ef-4506-8d68-5803bed81380\") " pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:25:31.833982 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:31.833919    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f62f64-a4ef-4506-8d68-5803bed81380-proxy-tls\") pod \"message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26\" (UID: \"d8f62f64-a4ef-4506-8d68-5803bed81380\") " pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:25:31.934714 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:31.934672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f62f64-a4ef-4506-8d68-5803bed81380-proxy-tls\") pod \"message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26\" (UID: \"d8f62f64-a4ef-4506-8d68-5803bed81380\") " pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:25:31.934938 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:31.934730    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l857v\" (UniqueName: \"kubernetes.io/projected/d8f62f64-a4ef-4506-8d68-5803bed81380-kube-api-access-l857v\") pod \"message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26\" (UID: \"d8f62f64-a4ef-4506-8d68-5803bed81380\") " pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:25:31.934938 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:31.934755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8f62f64-a4ef-4506-8d68-5803bed81380-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26\" (UID: \"d8f62f64-a4ef-4506-8d68-5803bed81380\") " pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:25:31.935516 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:31.935426    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8f62f64-a4ef-4506-8d68-5803bed81380-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26\" (UID: \"d8f62f64-a4ef-4506-8d68-5803bed81380\") " pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:25:31.937330 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:31.937308    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f62f64-a4ef-4506-8d68-5803bed81380-proxy-tls\") pod \"message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26\" (UID: \"d8f62f64-a4ef-4506-8d68-5803bed81380\") " pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:25:31.942972 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:31.942944    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l857v\" (UniqueName: \"kubernetes.io/projected/d8f62f64-a4ef-4506-8d68-5803bed81380-kube-api-access-l857v\") pod \"message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26\" (UID: \"d8f62f64-a4ef-4506-8d68-5803bed81380\") " pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:25:32.142300 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:32.142269    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:25:32.414358 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:32.414186    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"]
Sep 17 23:25:32.417035 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:25:32.417008    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f62f64_a4ef_4506_8d68_5803bed81380.slice/crio-28ca5aff3db8dd6aa4036e172a0903fd41ea3ca5515e17e536c00dc125f7e3a0 WatchSource:0}: Error finding container 28ca5aff3db8dd6aa4036e172a0903fd41ea3ca5515e17e536c00dc125f7e3a0: Status 404 returned error can't find the container with id 28ca5aff3db8dd6aa4036e172a0903fd41ea3ca5515e17e536c00dc125f7e3a0
Sep 17 23:25:32.994902 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:32.994866    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26" event={"ID":"d8f62f64-a4ef-4506-8d68-5803bed81380","Type":"ContainerStarted","Data":"28ca5aff3db8dd6aa4036e172a0903fd41ea3ca5515e17e536c00dc125f7e3a0"}
Sep 17 23:25:33.903138 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:33.903099    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 23:25:34.001621 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:34.001533    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26" event={"ID":"d8f62f64-a4ef-4506-8d68-5803bed81380","Type":"ContainerStarted","Data":"f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1"}
Sep 17 23:25:34.001621 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:34.001599    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26" event={"ID":"d8f62f64-a4ef-4506-8d68-5803bed81380","Type":"ContainerStarted","Data":"a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520"}
Sep 17 23:25:34.002033 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:34.001652    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:25:34.020592 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:34.020534    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26" podStartSLOduration=1.773300033 podStartE2EDuration="3.020516783s" totalImagesPullingTime="1.24721675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:34.02047181 +0000 UTC m=+888.707544386" watchObservedRunningTime="2026-09-17 23:25:34.020516783 +0000 UTC m=+888.707589359"
Sep 17 23:25:35.005364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:35.005332    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:25:35.007393 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:35.007371    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:25:42.017205 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:42.017172    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:25:43.903311 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:43.903271    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 23:25:53.903606 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:25:53.903564    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 23:26:03.903112 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:03.903072    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 23:26:13.902787 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:13.902743    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 17 23:26:23.903598 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:23.903568    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:26:46.849046 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:46.849000    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"]
Sep 17 23:26:46.852914 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:46.852894    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:46.855554 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:46.855523    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-67d4dc-predictor-serving-cert\""
Sep 17 23:26:46.855678 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:46.855564    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-67d4dc\""
Sep 17 23:26:46.855678 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:46.855639    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-67d4dc-dockercfg-456mx\""
Sep 17 23:26:46.855678 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:46.855642    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-67d4dc-kube-rbac-proxy-sar-config\""
Sep 17 23:26:46.862857 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:46.862813    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"]
Sep 17 23:26:46.953054 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:46.953017    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59a62353-6bd2-4021-b180-d3f3b3d03c25-kserve-provision-location\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:46.953255 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:46.953082    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59a62353-6bd2-4021-b180-d3f3b3d03c25-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:46.953255 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:46.953114    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59a62353-6bd2-4021-b180-d3f3b3d03c25-proxy-tls\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:46.953255 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:46.953149    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59a62353-6bd2-4021-b180-d3f3b3d03c25-cabundle-cert\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:46.953255 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:46.953235    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz96c\" (UniqueName: \"kubernetes.io/projected/59a62353-6bd2-4021-b180-d3f3b3d03c25-kube-api-access-rz96c\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:47.054276 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:47.054235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59a62353-6bd2-4021-b180-d3f3b3d03c25-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:47.054276 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:47.054285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59a62353-6bd2-4021-b180-d3f3b3d03c25-proxy-tls\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:47.054531 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:47.054322    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59a62353-6bd2-4021-b180-d3f3b3d03c25-cabundle-cert\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:47.054531 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:47.054353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz96c\" (UniqueName: \"kubernetes.io/projected/59a62353-6bd2-4021-b180-d3f3b3d03c25-kube-api-access-rz96c\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:47.054531 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:47.054404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59a62353-6bd2-4021-b180-d3f3b3d03c25-kserve-provision-location\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:47.054531 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:26:47.054481    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-serving-cert: secret "isvc-secondary-67d4dc-predictor-serving-cert" not found
Sep 17 23:26:47.054707 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:26:47.054548    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a62353-6bd2-4021-b180-d3f3b3d03c25-proxy-tls podName:59a62353-6bd2-4021-b180-d3f3b3d03c25 nodeName:}" failed. No retries permitted until 2026-09-17 23:26:47.554527689 +0000 UTC m=+962.241600260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/59a62353-6bd2-4021-b180-d3f3b3d03c25-proxy-tls") pod "isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd" (UID: "59a62353-6bd2-4021-b180-d3f3b3d03c25") : secret "isvc-secondary-67d4dc-predictor-serving-cert" not found
Sep 17 23:26:47.054819 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:47.054795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59a62353-6bd2-4021-b180-d3f3b3d03c25-kserve-provision-location\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:47.055141 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:47.055118    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59a62353-6bd2-4021-b180-d3f3b3d03c25-cabundle-cert\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:47.055190 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:47.055118    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59a62353-6bd2-4021-b180-d3f3b3d03c25-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:47.066991 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:47.066968    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz96c\" (UniqueName: \"kubernetes.io/projected/59a62353-6bd2-4021-b180-d3f3b3d03c25-kube-api-access-rz96c\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:47.559728 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:47.559694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59a62353-6bd2-4021-b180-d3f3b3d03c25-proxy-tls\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:47.562295 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:47.562263    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59a62353-6bd2-4021-b180-d3f3b3d03c25-proxy-tls\") pod \"isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") " pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:47.766255 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:47.766220    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-67d4dc-dockercfg-456mx\""
Sep 17 23:26:47.774363 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:47.774337    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:26:48.047808 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:48.047782    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"]
Sep 17 23:26:48.050079 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:26:48.050051    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a62353_6bd2_4021_b180_d3f3b3d03c25.slice/crio-fef407b670d86248bed9d6e52d23695147fbc365d2ecd23bcc23fc2f4ebffe81 WatchSource:0}: Error finding container fef407b670d86248bed9d6e52d23695147fbc365d2ecd23bcc23fc2f4ebffe81: Status 404 returned error can't find the container with id fef407b670d86248bed9d6e52d23695147fbc365d2ecd23bcc23fc2f4ebffe81
Sep 17 23:26:48.054396 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:48.054374    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-67d4dc\""
Sep 17 23:26:48.298071 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:48.298029    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd" event={"ID":"59a62353-6bd2-4021-b180-d3f3b3d03c25","Type":"ContainerStarted","Data":"681f2feb611d2785369e1516ad41bce38400073e7635a9c6788f192d1db1175e"}
Sep 17 23:26:48.298267 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:48.298089    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd" event={"ID":"59a62353-6bd2-4021-b180-d3f3b3d03c25","Type":"ContainerStarted","Data":"fef407b670d86248bed9d6e52d23695147fbc365d2ecd23bcc23fc2f4ebffe81"}
Sep 17 23:26:53.318923 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:53.318893    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd_59a62353-6bd2-4021-b180-d3f3b3d03c25/storage-initializer/0.log"
Sep 17 23:26:53.319324 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:53.318930    2582 generic.go:356] "Generic (PLEG): container finished" podID="59a62353-6bd2-4021-b180-d3f3b3d03c25" containerID="681f2feb611d2785369e1516ad41bce38400073e7635a9c6788f192d1db1175e" exitCode=1
Sep 17 23:26:53.319324 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:53.318984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd" event={"ID":"59a62353-6bd2-4021-b180-d3f3b3d03c25","Type":"ContainerDied","Data":"681f2feb611d2785369e1516ad41bce38400073e7635a9c6788f192d1db1175e"}
Sep 17 23:26:54.324387 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:54.324348    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd_59a62353-6bd2-4021-b180-d3f3b3d03c25/storage-initializer/0.log"
Sep 17 23:26:54.324806 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:54.324465    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd" event={"ID":"59a62353-6bd2-4021-b180-d3f3b3d03c25","Type":"ContainerStarted","Data":"e6cacb4c4e4b59d79e61d8f86a37ca868beff67489b7f0b5b3cd066c50c36441"}
Sep 17 23:26:58.341582 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:58.341499    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd_59a62353-6bd2-4021-b180-d3f3b3d03c25/storage-initializer/1.log"
Sep 17 23:26:58.341978 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:58.341894    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd_59a62353-6bd2-4021-b180-d3f3b3d03c25/storage-initializer/0.log"
Sep 17 23:26:58.341978 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:58.341922    2582 generic.go:356] "Generic (PLEG): container finished" podID="59a62353-6bd2-4021-b180-d3f3b3d03c25" containerID="e6cacb4c4e4b59d79e61d8f86a37ca868beff67489b7f0b5b3cd066c50c36441" exitCode=1
Sep 17 23:26:58.342068 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:58.341989    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd" event={"ID":"59a62353-6bd2-4021-b180-d3f3b3d03c25","Type":"ContainerDied","Data":"e6cacb4c4e4b59d79e61d8f86a37ca868beff67489b7f0b5b3cd066c50c36441"}
Sep 17 23:26:58.342068 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:58.342046    2582 scope.go:117] "RemoveContainer" containerID="681f2feb611d2785369e1516ad41bce38400073e7635a9c6788f192d1db1175e"
Sep 17 23:26:58.342478 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:58.342453    2582 scope.go:117] "RemoveContainer" containerID="681f2feb611d2785369e1516ad41bce38400073e7635a9c6788f192d1db1175e"
Sep 17 23:26:58.353378 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:26:58.353356    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd_kserve-ci-e2e-test(59a62353-6bd2-4021-b180-d3f3b3d03c25)\"" pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd" podUID="59a62353-6bd2-4021-b180-d3f3b3d03c25"
Sep 17 23:26:58.353584 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:26:58.353556    2582 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd_kserve-ci-e2e-test_59a62353-6bd2-4021-b180-d3f3b3d03c25_0 in pod sandbox fef407b670d86248bed9d6e52d23695147fbc365d2ecd23bcc23fc2f4ebffe81 from index: no such id: '681f2feb611d2785369e1516ad41bce38400073e7635a9c6788f192d1db1175e'" containerID="681f2feb611d2785369e1516ad41bce38400073e7635a9c6788f192d1db1175e"
Sep 17 23:26:58.353639 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:58.353595    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"681f2feb611d2785369e1516ad41bce38400073e7635a9c6788f192d1db1175e"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd_kserve-ci-e2e-test_59a62353-6bd2-4021-b180-d3f3b3d03c25_0 in pod sandbox fef407b670d86248bed9d6e52d23695147fbc365d2ecd23bcc23fc2f4ebffe81 from index: no such id: '681f2feb611d2785369e1516ad41bce38400073e7635a9c6788f192d1db1175e'"
Sep 17 23:26:59.347647 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:26:59.347615    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd_59a62353-6bd2-4021-b180-d3f3b3d03c25/storage-initializer/1.log"
Sep 17 23:27:04.904502 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:04.904468    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"]
Sep 17 23:27:04.937622 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:04.937591    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"]
Sep 17 23:27:04.937949 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:04.937900    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kserve-container" containerID="cri-o://d0c814b33f40cd930893d5c2e300fff5884fc7db7d76946934fa375ee4581f55" gracePeriod=30
Sep 17 23:27:04.938042 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:04.937993    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kube-rbac-proxy" containerID="cri-o://99463aa085e18b122d42f8f721c0718732620a116ef0eab4f9ca945c4294a6bb" gracePeriod=30
Sep 17 23:27:05.058053 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.058032    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd_59a62353-6bd2-4021-b180-d3f3b3d03c25/storage-initializer/1.log"
Sep 17 23:27:05.060040 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.060025    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:27:05.205454 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.205338    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59a62353-6bd2-4021-b180-d3f3b3d03c25-kube-rbac-proxy-sar-config\") pod \"59a62353-6bd2-4021-b180-d3f3b3d03c25\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") "
Sep 17 23:27:05.205454 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.205414    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59a62353-6bd2-4021-b180-d3f3b3d03c25-kserve-provision-location\") pod \"59a62353-6bd2-4021-b180-d3f3b3d03c25\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") "
Sep 17 23:27:05.205454 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.205446    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59a62353-6bd2-4021-b180-d3f3b3d03c25-cabundle-cert\") pod \"59a62353-6bd2-4021-b180-d3f3b3d03c25\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") "
Sep 17 23:27:05.205762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.205476    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz96c\" (UniqueName: \"kubernetes.io/projected/59a62353-6bd2-4021-b180-d3f3b3d03c25-kube-api-access-rz96c\") pod \"59a62353-6bd2-4021-b180-d3f3b3d03c25\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") "
Sep 17 23:27:05.205762 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.205516    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59a62353-6bd2-4021-b180-d3f3b3d03c25-proxy-tls\") pod \"59a62353-6bd2-4021-b180-d3f3b3d03c25\" (UID: \"59a62353-6bd2-4021-b180-d3f3b3d03c25\") "
Sep 17 23:27:05.205908 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.205759    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a62353-6bd2-4021-b180-d3f3b3d03c25-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "59a62353-6bd2-4021-b180-d3f3b3d03c25" (UID: "59a62353-6bd2-4021-b180-d3f3b3d03c25"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:05.205908 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.205774    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a62353-6bd2-4021-b180-d3f3b3d03c25-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59a62353-6bd2-4021-b180-d3f3b3d03c25" (UID: "59a62353-6bd2-4021-b180-d3f3b3d03c25"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:05.205990 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.205908    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a62353-6bd2-4021-b180-d3f3b3d03c25-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "59a62353-6bd2-4021-b180-d3f3b3d03c25" (UID: "59a62353-6bd2-4021-b180-d3f3b3d03c25"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:05.207847 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.207807    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a62353-6bd2-4021-b180-d3f3b3d03c25-kube-api-access-rz96c" (OuterVolumeSpecName: "kube-api-access-rz96c") pod "59a62353-6bd2-4021-b180-d3f3b3d03c25" (UID: "59a62353-6bd2-4021-b180-d3f3b3d03c25"). InnerVolumeSpecName "kube-api-access-rz96c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:05.207958 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.207939    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a62353-6bd2-4021-b180-d3f3b3d03c25-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "59a62353-6bd2-4021-b180-d3f3b3d03c25" (UID: "59a62353-6bd2-4021-b180-d3f3b3d03c25"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:27:05.306722 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.306673    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59a62353-6bd2-4021-b180-d3f3b3d03c25-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:05.306722 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.306715    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59a62353-6bd2-4021-b180-d3f3b3d03c25-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:05.306722 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.306725    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz96c\" (UniqueName: \"kubernetes.io/projected/59a62353-6bd2-4021-b180-d3f3b3d03c25-kube-api-access-rz96c\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:05.306722 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.306736    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59a62353-6bd2-4021-b180-d3f3b3d03c25-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:05.307064 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.306746    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59a62353-6bd2-4021-b180-d3f3b3d03c25-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:05.374321 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.374296    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd_59a62353-6bd2-4021-b180-d3f3b3d03c25/storage-initializer/1.log"
Sep 17 23:27:05.374497 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.374415    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"
Sep 17 23:27:05.374559 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.374420    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd" event={"ID":"59a62353-6bd2-4021-b180-d3f3b3d03c25","Type":"ContainerDied","Data":"fef407b670d86248bed9d6e52d23695147fbc365d2ecd23bcc23fc2f4ebffe81"}
Sep 17 23:27:05.374601 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.374573    2582 scope.go:117] "RemoveContainer" containerID="e6cacb4c4e4b59d79e61d8f86a37ca868beff67489b7f0b5b3cd066c50c36441"
Sep 17 23:27:05.376919 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.376892    2582 generic.go:356] "Generic (PLEG): container finished" podID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerID="99463aa085e18b122d42f8f721c0718732620a116ef0eab4f9ca945c4294a6bb" exitCode=2
Sep 17 23:27:05.377033 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.376928    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" event={"ID":"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa","Type":"ContainerDied","Data":"99463aa085e18b122d42f8f721c0718732620a116ef0eab4f9ca945c4294a6bb"}
Sep 17 23:27:05.418715 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.418682    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"]
Sep 17 23:27:05.423648 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:05.423626    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-67d4dc-predictor-8548b55dd5-chvdd"]
Sep 17 23:27:07.399108 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:07.399072    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59a62353-6bd2-4021-b180-d3f3b3d03c25" path="/var/lib/kubelet/pods/59a62353-6bd2-4021-b180-d3f3b3d03c25/volumes"
Sep 17 23:27:08.897638 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:08.897588    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 17 23:27:10.080365 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.080337    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:27:10.196617 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.196576    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"]
Sep 17 23:27:10.196905 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.196892    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="storage-initializer"
Sep 17 23:27:10.196971 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.196906    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="storage-initializer"
Sep 17 23:27:10.196971 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.196914    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a62353-6bd2-4021-b180-d3f3b3d03c25" containerName="storage-initializer"
Sep 17 23:27:10.196971 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.196921    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a62353-6bd2-4021-b180-d3f3b3d03c25" containerName="storage-initializer"
Sep 17 23:27:10.196971 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.196938    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a62353-6bd2-4021-b180-d3f3b3d03c25" containerName="storage-initializer"
Sep 17 23:27:10.196971 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.196944    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a62353-6bd2-4021-b180-d3f3b3d03c25" containerName="storage-initializer"
Sep 17 23:27:10.196971 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.196951    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kserve-container"
Sep 17 23:27:10.196971 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.196957    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kserve-container"
Sep 17 23:27:10.196971 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.196965    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kube-rbac-proxy"
Sep 17 23:27:10.196971 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.196970    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kube-rbac-proxy"
Sep 17 23:27:10.197225 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.197022    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kube-rbac-proxy"
Sep 17 23:27:10.197225 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.197030    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a62353-6bd2-4021-b180-d3f3b3d03c25" containerName="storage-initializer"
Sep 17 23:27:10.197225 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.197036    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a62353-6bd2-4021-b180-d3f3b3d03c25" containerName="storage-initializer"
Sep 17 23:27:10.197225 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.197042    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerName="kserve-container"
Sep 17 23:27:10.200070 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.200049    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.203370 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.203324    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-edaf65-predictor-serving-cert\""
Sep 17 23:27:10.203589 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.203569    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-edaf65\""
Sep 17 23:27:10.203753 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.203728    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-edaf65-dockercfg-tktlg\""
Sep 17 23:27:10.203868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.203732    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-edaf65-kube-rbac-proxy-sar-config\""
Sep 17 23:27:10.213647 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.213621    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"]
Sep 17 23:27:10.251494 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.251453    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bf5p8\" (UniqueName: \"kubernetes.io/projected/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kube-api-access-bf5p8\") pod \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") "
Sep 17 23:27:10.251693 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.251500    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kube-rbac-proxy-sar-config\") pod \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") "
Sep 17 23:27:10.251693 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.251548    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-proxy-tls\") pod \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") "
Sep 17 23:27:10.251693 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.251578    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-cabundle-cert\") pod \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") "
Sep 17 23:27:10.251693 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.251609    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kserve-provision-location\") pod \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\" (UID: \"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa\") "
Sep 17 23:27:10.251986 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.251958    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" (UID: "35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:10.252039 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.251965    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" (UID: "35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:10.252158 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.252129    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" (UID: "35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:10.253998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.253970    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" (UID: "35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:27:10.254085 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.254026    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kube-api-access-bf5p8" (OuterVolumeSpecName: "kube-api-access-bf5p8") pod "35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" (UID: "35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa"). InnerVolumeSpecName "kube-api-access-bf5p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:10.352681 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.352644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec67762d-366b-4532-b6af-50b0cbba5bc8-cabundle-cert\") pod \"isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") " pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.352867 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.352706    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfbgr\" (UniqueName: \"kubernetes.io/projected/ec67762d-366b-4532-b6af-50b0cbba5bc8-kube-api-access-lfbgr\") pod \"isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") " pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.352867 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.352769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec67762d-366b-4532-b6af-50b0cbba5bc8-proxy-tls\") pod \"isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") " pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.352981 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.352854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec67762d-366b-4532-b6af-50b0cbba5bc8-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") " pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.352981 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.352897    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec67762d-366b-4532-b6af-50b0cbba5bc8-kserve-provision-location\") pod \"isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") " pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.352981 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.352959    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf5p8\" (UniqueName: \"kubernetes.io/projected/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kube-api-access-bf5p8\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:10.352981 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.352969    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:10.353117 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.352980    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:10.353117 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.352993    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:10.353117 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.353002    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:10.399388 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.399359    2582 generic.go:356] "Generic (PLEG): container finished" podID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" containerID="d0c814b33f40cd930893d5c2e300fff5884fc7db7d76946934fa375ee4581f55" exitCode=0
Sep 17 23:27:10.399559 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.399422    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" event={"ID":"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa","Type":"ContainerDied","Data":"d0c814b33f40cd930893d5c2e300fff5884fc7db7d76946934fa375ee4581f55"}
Sep 17 23:27:10.399559 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.399439    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"
Sep 17 23:27:10.399559 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.399468    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t" event={"ID":"35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa","Type":"ContainerDied","Data":"62b3ecfc5d0fe927b9f9f575e949c9b2041630371cb1e4dbc83837d789cddf82"}
Sep 17 23:27:10.399559 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.399497    2582 scope.go:117] "RemoveContainer" containerID="99463aa085e18b122d42f8f721c0718732620a116ef0eab4f9ca945c4294a6bb"
Sep 17 23:27:10.408062 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.408042    2582 scope.go:117] "RemoveContainer" containerID="d0c814b33f40cd930893d5c2e300fff5884fc7db7d76946934fa375ee4581f55"
Sep 17 23:27:10.415551 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.415523    2582 scope.go:117] "RemoveContainer" containerID="6f2b18e28f40ef87c570b90c1fcad4b2f19f873bbf864c45cdb5cfdb69b7ca77"
Sep 17 23:27:10.422918 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.422901    2582 scope.go:117] "RemoveContainer" containerID="99463aa085e18b122d42f8f721c0718732620a116ef0eab4f9ca945c4294a6bb"
Sep 17 23:27:10.423184 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:27:10.423165    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99463aa085e18b122d42f8f721c0718732620a116ef0eab4f9ca945c4294a6bb\": container with ID starting with 99463aa085e18b122d42f8f721c0718732620a116ef0eab4f9ca945c4294a6bb not found: ID does not exist" containerID="99463aa085e18b122d42f8f721c0718732620a116ef0eab4f9ca945c4294a6bb"
Sep 17 23:27:10.423231 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.423193    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99463aa085e18b122d42f8f721c0718732620a116ef0eab4f9ca945c4294a6bb"} err="failed to get container status \"99463aa085e18b122d42f8f721c0718732620a116ef0eab4f9ca945c4294a6bb\": rpc error: code = NotFound desc = could not find container \"99463aa085e18b122d42f8f721c0718732620a116ef0eab4f9ca945c4294a6bb\": container with ID starting with 99463aa085e18b122d42f8f721c0718732620a116ef0eab4f9ca945c4294a6bb not found: ID does not exist"
Sep 17 23:27:10.423231 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.423211    2582 scope.go:117] "RemoveContainer" containerID="d0c814b33f40cd930893d5c2e300fff5884fc7db7d76946934fa375ee4581f55"
Sep 17 23:27:10.423463 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:27:10.423445    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c814b33f40cd930893d5c2e300fff5884fc7db7d76946934fa375ee4581f55\": container with ID starting with d0c814b33f40cd930893d5c2e300fff5884fc7db7d76946934fa375ee4581f55 not found: ID does not exist" containerID="d0c814b33f40cd930893d5c2e300fff5884fc7db7d76946934fa375ee4581f55"
Sep 17 23:27:10.423503 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.423469    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c814b33f40cd930893d5c2e300fff5884fc7db7d76946934fa375ee4581f55"} err="failed to get container status \"d0c814b33f40cd930893d5c2e300fff5884fc7db7d76946934fa375ee4581f55\": rpc error: code = NotFound desc = could not find container \"d0c814b33f40cd930893d5c2e300fff5884fc7db7d76946934fa375ee4581f55\": container with ID starting with d0c814b33f40cd930893d5c2e300fff5884fc7db7d76946934fa375ee4581f55 not found: ID does not exist"
Sep 17 23:27:10.423503 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.423485    2582 scope.go:117] "RemoveContainer" containerID="6f2b18e28f40ef87c570b90c1fcad4b2f19f873bbf864c45cdb5cfdb69b7ca77"
Sep 17 23:27:10.423761 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:27:10.423731    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2b18e28f40ef87c570b90c1fcad4b2f19f873bbf864c45cdb5cfdb69b7ca77\": container with ID starting with 6f2b18e28f40ef87c570b90c1fcad4b2f19f873bbf864c45cdb5cfdb69b7ca77 not found: ID does not exist" containerID="6f2b18e28f40ef87c570b90c1fcad4b2f19f873bbf864c45cdb5cfdb69b7ca77"
Sep 17 23:27:10.423806 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.423769    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2b18e28f40ef87c570b90c1fcad4b2f19f873bbf864c45cdb5cfdb69b7ca77"} err="failed to get container status \"6f2b18e28f40ef87c570b90c1fcad4b2f19f873bbf864c45cdb5cfdb69b7ca77\": rpc error: code = NotFound desc = could not find container \"6f2b18e28f40ef87c570b90c1fcad4b2f19f873bbf864c45cdb5cfdb69b7ca77\": container with ID starting with 6f2b18e28f40ef87c570b90c1fcad4b2f19f873bbf864c45cdb5cfdb69b7ca77 not found: ID does not exist"
Sep 17 23:27:10.427722 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.427699    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"]
Sep 17 23:27:10.430276 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.430251    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-67d4dc-predictor-774f887c4b-h476t"]
Sep 17 23:27:10.454005 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.453948    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec67762d-366b-4532-b6af-50b0cbba5bc8-proxy-tls\") pod \"isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") " pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.454005 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.453987    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec67762d-366b-4532-b6af-50b0cbba5bc8-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") " pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.454187 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.454015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec67762d-366b-4532-b6af-50b0cbba5bc8-kserve-provision-location\") pod \"isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") " pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.454187 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.454047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec67762d-366b-4532-b6af-50b0cbba5bc8-cabundle-cert\") pod \"isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") " pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.454187 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.454076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbgr\" (UniqueName: \"kubernetes.io/projected/ec67762d-366b-4532-b6af-50b0cbba5bc8-kube-api-access-lfbgr\") pod \"isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") " pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.454509 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.454490    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec67762d-366b-4532-b6af-50b0cbba5bc8-kserve-provision-location\") pod \"isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") " pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.454734 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.454717    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec67762d-366b-4532-b6af-50b0cbba5bc8-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") " pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.454776 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.454722    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec67762d-366b-4532-b6af-50b0cbba5bc8-cabundle-cert\") pod \"isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") " pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.456584 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.456569    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec67762d-366b-4532-b6af-50b0cbba5bc8-proxy-tls\") pod \"isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") " pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.462408 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.462389    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfbgr\" (UniqueName: \"kubernetes.io/projected/ec67762d-366b-4532-b6af-50b0cbba5bc8-kube-api-access-lfbgr\") pod \"isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") " pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.510479 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.510440    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:10.790575 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:10.790552    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"]
Sep 17 23:27:10.792991 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:27:10.792956    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec67762d_366b_4532_b6af_50b0cbba5bc8.slice/crio-f444a45edb76a28da95ba09e17d9a4c6c6e99c1613c02fd04fc850442a56acee WatchSource:0}: Error finding container f444a45edb76a28da95ba09e17d9a4c6c6e99c1613c02fd04fc850442a56acee: Status 404 returned error can't find the container with id f444a45edb76a28da95ba09e17d9a4c6c6e99c1613c02fd04fc850442a56acee
Sep 17 23:27:11.393995 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:11.393964    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa" path="/var/lib/kubelet/pods/35541dae-4ee3-4b82-8b5d-fbe6d5e1dbaa/volumes"
Sep 17 23:27:11.406670 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:11.406637    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf" event={"ID":"ec67762d-366b-4532-b6af-50b0cbba5bc8","Type":"ContainerStarted","Data":"145f5428c4351a7b000fcc2eb9fb4a37ce8a7cb2a644320a41b9b56316664a5b"}
Sep 17 23:27:11.406871 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:11.406697    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf" event={"ID":"ec67762d-366b-4532-b6af-50b0cbba5bc8","Type":"ContainerStarted","Data":"f444a45edb76a28da95ba09e17d9a4c6c6e99c1613c02fd04fc850442a56acee"}
Sep 17 23:27:14.419516 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:14.419488    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf_ec67762d-366b-4532-b6af-50b0cbba5bc8/storage-initializer/0.log"
Sep 17 23:27:14.419944 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:14.419532    2582 generic.go:356] "Generic (PLEG): container finished" podID="ec67762d-366b-4532-b6af-50b0cbba5bc8" containerID="145f5428c4351a7b000fcc2eb9fb4a37ce8a7cb2a644320a41b9b56316664a5b" exitCode=1
Sep 17 23:27:14.419944 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:14.419613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf" event={"ID":"ec67762d-366b-4532-b6af-50b0cbba5bc8","Type":"ContainerDied","Data":"145f5428c4351a7b000fcc2eb9fb4a37ce8a7cb2a644320a41b9b56316664a5b"}
Sep 17 23:27:15.135164 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:15.135123    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"]
Sep 17 23:27:15.425817 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:15.425734    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf_ec67762d-366b-4532-b6af-50b0cbba5bc8/storage-initializer/0.log"
Sep 17 23:27:15.425817 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:15.425782    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf" event={"ID":"ec67762d-366b-4532-b6af-50b0cbba5bc8","Type":"ContainerStarted","Data":"e9c9363e31d11ee19ab3ce2f3b390ed6fa2af7f41d2a2cbc6e14feae810fb08f"}
Sep 17 23:27:15.426306 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:15.425979    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf" podUID="ec67762d-366b-4532-b6af-50b0cbba5bc8" containerName="storage-initializer" containerID="cri-o://e9c9363e31d11ee19ab3ce2f3b390ed6fa2af7f41d2a2cbc6e14feae810fb08f" gracePeriod=30
Sep 17 23:27:18.367152 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.367128    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf_ec67762d-366b-4532-b6af-50b0cbba5bc8/storage-initializer/1.log"
Sep 17 23:27:18.367538 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.367523    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf_ec67762d-366b-4532-b6af-50b0cbba5bc8/storage-initializer/0.log"
Sep 17 23:27:18.369866 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.369850    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:18.416821 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.416787    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec67762d-366b-4532-b6af-50b0cbba5bc8-cabundle-cert\") pod \"ec67762d-366b-4532-b6af-50b0cbba5bc8\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") "
Sep 17 23:27:18.416821 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.416824    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec67762d-366b-4532-b6af-50b0cbba5bc8-proxy-tls\") pod \"ec67762d-366b-4532-b6af-50b0cbba5bc8\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") "
Sep 17 23:27:18.417091 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.416868    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfbgr\" (UniqueName: \"kubernetes.io/projected/ec67762d-366b-4532-b6af-50b0cbba5bc8-kube-api-access-lfbgr\") pod \"ec67762d-366b-4532-b6af-50b0cbba5bc8\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") "
Sep 17 23:27:18.417091 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.416892    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec67762d-366b-4532-b6af-50b0cbba5bc8-kserve-provision-location\") pod \"ec67762d-366b-4532-b6af-50b0cbba5bc8\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") "
Sep 17 23:27:18.417091 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.416922    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec67762d-366b-4532-b6af-50b0cbba5bc8-kube-rbac-proxy-sar-config\") pod \"ec67762d-366b-4532-b6af-50b0cbba5bc8\" (UID: \"ec67762d-366b-4532-b6af-50b0cbba5bc8\") "
Sep 17 23:27:18.417248 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.417173    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec67762d-366b-4532-b6af-50b0cbba5bc8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ec67762d-366b-4532-b6af-50b0cbba5bc8" (UID: "ec67762d-366b-4532-b6af-50b0cbba5bc8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:18.417248 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.417184    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec67762d-366b-4532-b6af-50b0cbba5bc8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec67762d-366b-4532-b6af-50b0cbba5bc8" (UID: "ec67762d-366b-4532-b6af-50b0cbba5bc8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:18.417540 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.417503    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec67762d-366b-4532-b6af-50b0cbba5bc8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec67762d-366b-4532-b6af-50b0cbba5bc8" (UID: "ec67762d-366b-4532-b6af-50b0cbba5bc8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:18.419293 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.419270    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec67762d-366b-4532-b6af-50b0cbba5bc8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec67762d-366b-4532-b6af-50b0cbba5bc8" (UID: "ec67762d-366b-4532-b6af-50b0cbba5bc8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:27:18.419390 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.419272    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec67762d-366b-4532-b6af-50b0cbba5bc8-kube-api-access-lfbgr" (OuterVolumeSpecName: "kube-api-access-lfbgr") pod "ec67762d-366b-4532-b6af-50b0cbba5bc8" (UID: "ec67762d-366b-4532-b6af-50b0cbba5bc8"). InnerVolumeSpecName "kube-api-access-lfbgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:18.439926 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.439863    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf_ec67762d-366b-4532-b6af-50b0cbba5bc8/storage-initializer/1.log"
Sep 17 23:27:18.440256 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.440241    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf_ec67762d-366b-4532-b6af-50b0cbba5bc8/storage-initializer/0.log"
Sep 17 23:27:18.440310 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.440276    2582 generic.go:356] "Generic (PLEG): container finished" podID="ec67762d-366b-4532-b6af-50b0cbba5bc8" containerID="e9c9363e31d11ee19ab3ce2f3b390ed6fa2af7f41d2a2cbc6e14feae810fb08f" exitCode=1
Sep 17 23:27:18.440354 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.440337    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"
Sep 17 23:27:18.440390 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.440348    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf" event={"ID":"ec67762d-366b-4532-b6af-50b0cbba5bc8","Type":"ContainerDied","Data":"e9c9363e31d11ee19ab3ce2f3b390ed6fa2af7f41d2a2cbc6e14feae810fb08f"}
Sep 17 23:27:18.440434 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.440406    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf" event={"ID":"ec67762d-366b-4532-b6af-50b0cbba5bc8","Type":"ContainerDied","Data":"f444a45edb76a28da95ba09e17d9a4c6c6e99c1613c02fd04fc850442a56acee"}
Sep 17 23:27:18.440434 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.440432    2582 scope.go:117] "RemoveContainer" containerID="e9c9363e31d11ee19ab3ce2f3b390ed6fa2af7f41d2a2cbc6e14feae810fb08f"
Sep 17 23:27:18.452754 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.452731    2582 scope.go:117] "RemoveContainer" containerID="145f5428c4351a7b000fcc2eb9fb4a37ce8a7cb2a644320a41b9b56316664a5b"
Sep 17 23:27:18.460855 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.460823    2582 scope.go:117] "RemoveContainer" containerID="e9c9363e31d11ee19ab3ce2f3b390ed6fa2af7f41d2a2cbc6e14feae810fb08f"
Sep 17 23:27:18.461121 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:27:18.461102    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c9363e31d11ee19ab3ce2f3b390ed6fa2af7f41d2a2cbc6e14feae810fb08f\": container with ID starting with e9c9363e31d11ee19ab3ce2f3b390ed6fa2af7f41d2a2cbc6e14feae810fb08f not found: ID does not exist" containerID="e9c9363e31d11ee19ab3ce2f3b390ed6fa2af7f41d2a2cbc6e14feae810fb08f"
Sep 17 23:27:18.461169 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.461129    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c9363e31d11ee19ab3ce2f3b390ed6fa2af7f41d2a2cbc6e14feae810fb08f"} err="failed to get container status \"e9c9363e31d11ee19ab3ce2f3b390ed6fa2af7f41d2a2cbc6e14feae810fb08f\": rpc error: code = NotFound desc = could not find container \"e9c9363e31d11ee19ab3ce2f3b390ed6fa2af7f41d2a2cbc6e14feae810fb08f\": container with ID starting with e9c9363e31d11ee19ab3ce2f3b390ed6fa2af7f41d2a2cbc6e14feae810fb08f not found: ID does not exist"
Sep 17 23:27:18.461169 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.461145    2582 scope.go:117] "RemoveContainer" containerID="145f5428c4351a7b000fcc2eb9fb4a37ce8a7cb2a644320a41b9b56316664a5b"
Sep 17 23:27:18.461381 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:27:18.461361    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145f5428c4351a7b000fcc2eb9fb4a37ce8a7cb2a644320a41b9b56316664a5b\": container with ID starting with 145f5428c4351a7b000fcc2eb9fb4a37ce8a7cb2a644320a41b9b56316664a5b not found: ID does not exist" containerID="145f5428c4351a7b000fcc2eb9fb4a37ce8a7cb2a644320a41b9b56316664a5b"
Sep 17 23:27:18.461439 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.461389    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145f5428c4351a7b000fcc2eb9fb4a37ce8a7cb2a644320a41b9b56316664a5b"} err="failed to get container status \"145f5428c4351a7b000fcc2eb9fb4a37ce8a7cb2a644320a41b9b56316664a5b\": rpc error: code = NotFound desc = could not find container \"145f5428c4351a7b000fcc2eb9fb4a37ce8a7cb2a644320a41b9b56316664a5b\": container with ID starting with 145f5428c4351a7b000fcc2eb9fb4a37ce8a7cb2a644320a41b9b56316664a5b not found: ID does not exist"
Sep 17 23:27:18.482305 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.482279    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"]
Sep 17 23:27:18.486067 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.486039    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-edaf65-predictor-6d87d9675c-jm4vf"]
Sep 17 23:27:18.517801 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.517766    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec67762d-366b-4532-b6af-50b0cbba5bc8-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:18.517801 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.517794    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec67762d-366b-4532-b6af-50b0cbba5bc8-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:18.517801 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.517804    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfbgr\" (UniqueName: \"kubernetes.io/projected/ec67762d-366b-4532-b6af-50b0cbba5bc8-kube-api-access-lfbgr\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:18.518052 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.517814    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec67762d-366b-4532-b6af-50b0cbba5bc8-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:18.518052 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:18.517824    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec67762d-366b-4532-b6af-50b0cbba5bc8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:19.393626 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:19.393591    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec67762d-366b-4532-b6af-50b0cbba5bc8" path="/var/lib/kubelet/pods/ec67762d-366b-4532-b6af-50b0cbba5bc8/volumes"
Sep 17 23:27:20.367327 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.367294    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"]
Sep 17 23:27:20.367618 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.367607    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec67762d-366b-4532-b6af-50b0cbba5bc8" containerName="storage-initializer"
Sep 17 23:27:20.367658 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.367621    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec67762d-366b-4532-b6af-50b0cbba5bc8" containerName="storage-initializer"
Sep 17 23:27:20.367658 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.367637    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec67762d-366b-4532-b6af-50b0cbba5bc8" containerName="storage-initializer"
Sep 17 23:27:20.367658 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.367642    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec67762d-366b-4532-b6af-50b0cbba5bc8" containerName="storage-initializer"
Sep 17 23:27:20.367749 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.367686    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec67762d-366b-4532-b6af-50b0cbba5bc8" containerName="storage-initializer"
Sep 17 23:27:20.367792 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.367783    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec67762d-366b-4532-b6af-50b0cbba5bc8" containerName="storage-initializer"
Sep 17 23:27:20.372285 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.372264    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.374471 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.374447    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-35411-predictor-serving-cert\""
Sep 17 23:27:20.374768 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.374743    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:27:20.374868 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.374775    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-35411-kube-rbac-proxy-sar-config\""
Sep 17 23:27:20.374925 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.374864    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:27:20.381337 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.381314    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"]
Sep 17 23:27:20.429682 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.429644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a06322e6-cd3d-4665-9b2a-378aba1213dd-kserve-provision-location\") pod \"raw-sklearn-35411-predictor-65bc89658c-dth4r\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") " pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.430078 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.429718    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a06322e6-cd3d-4665-9b2a-378aba1213dd-proxy-tls\") pod \"raw-sklearn-35411-predictor-65bc89658c-dth4r\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") " pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.430078 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.429742    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a06322e6-cd3d-4665-9b2a-378aba1213dd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-35411-predictor-65bc89658c-dth4r\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") " pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.430078 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.429761    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a06322e6-cd3d-4665-9b2a-378aba1213dd-cabundle-cert\") pod \"raw-sklearn-35411-predictor-65bc89658c-dth4r\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") " pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.430078 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.429787    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htd6m\" (UniqueName: \"kubernetes.io/projected/a06322e6-cd3d-4665-9b2a-378aba1213dd-kube-api-access-htd6m\") pod \"raw-sklearn-35411-predictor-65bc89658c-dth4r\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") " pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.530689 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.530654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a06322e6-cd3d-4665-9b2a-378aba1213dd-kserve-provision-location\") pod \"raw-sklearn-35411-predictor-65bc89658c-dth4r\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") " pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.530882 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.530732    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a06322e6-cd3d-4665-9b2a-378aba1213dd-proxy-tls\") pod \"raw-sklearn-35411-predictor-65bc89658c-dth4r\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") " pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.530882 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.530760    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a06322e6-cd3d-4665-9b2a-378aba1213dd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-35411-predictor-65bc89658c-dth4r\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") " pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.530882 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.530781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a06322e6-cd3d-4665-9b2a-378aba1213dd-cabundle-cert\") pod \"raw-sklearn-35411-predictor-65bc89658c-dth4r\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") " pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.530882 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.530815    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htd6m\" (UniqueName: \"kubernetes.io/projected/a06322e6-cd3d-4665-9b2a-378aba1213dd-kube-api-access-htd6m\") pod \"raw-sklearn-35411-predictor-65bc89658c-dth4r\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") " pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.531160 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.531137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a06322e6-cd3d-4665-9b2a-378aba1213dd-kserve-provision-location\") pod \"raw-sklearn-35411-predictor-65bc89658c-dth4r\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") " pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.531417 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.531389    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a06322e6-cd3d-4665-9b2a-378aba1213dd-cabundle-cert\") pod \"raw-sklearn-35411-predictor-65bc89658c-dth4r\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") " pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.531522 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.531464    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a06322e6-cd3d-4665-9b2a-378aba1213dd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-35411-predictor-65bc89658c-dth4r\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") " pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.533412 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.533391    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a06322e6-cd3d-4665-9b2a-378aba1213dd-proxy-tls\") pod \"raw-sklearn-35411-predictor-65bc89658c-dth4r\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") " pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.539523 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.539503    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htd6m\" (UniqueName: \"kubernetes.io/projected/a06322e6-cd3d-4665-9b2a-378aba1213dd-kube-api-access-htd6m\") pod \"raw-sklearn-35411-predictor-65bc89658c-dth4r\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") " pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.682811 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.682776    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:20.960212 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:20.960184    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"]
Sep 17 23:27:20.962574 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:27:20.962538    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06322e6_cd3d_4665_9b2a_378aba1213dd.slice/crio-c943058d1d6028a6f86073b6fb4e73835a2e67aa8ece1ba0c4d5c60db6e7e9e6 WatchSource:0}: Error finding container c943058d1d6028a6f86073b6fb4e73835a2e67aa8ece1ba0c4d5c60db6e7e9e6: Status 404 returned error can't find the container with id c943058d1d6028a6f86073b6fb4e73835a2e67aa8ece1ba0c4d5c60db6e7e9e6
Sep 17 23:27:21.457550 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:21.457511    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" event={"ID":"a06322e6-cd3d-4665-9b2a-378aba1213dd","Type":"ContainerStarted","Data":"eee04f85479105907b821a142ab7ff312e5c72a5e33e52c69ca5aeb2d9f353e2"}
Sep 17 23:27:21.458041 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:21.457581    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" event={"ID":"a06322e6-cd3d-4665-9b2a-378aba1213dd","Type":"ContainerStarted","Data":"c943058d1d6028a6f86073b6fb4e73835a2e67aa8ece1ba0c4d5c60db6e7e9e6"}
Sep 17 23:27:25.475325 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:25.475289    2582 generic.go:356] "Generic (PLEG): container finished" podID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerID="eee04f85479105907b821a142ab7ff312e5c72a5e33e52c69ca5aeb2d9f353e2" exitCode=0
Sep 17 23:27:25.475732 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:25.475371    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" event={"ID":"a06322e6-cd3d-4665-9b2a-378aba1213dd","Type":"ContainerDied","Data":"eee04f85479105907b821a142ab7ff312e5c72a5e33e52c69ca5aeb2d9f353e2"}
Sep 17 23:27:26.482202 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:26.482161    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" event={"ID":"a06322e6-cd3d-4665-9b2a-378aba1213dd","Type":"ContainerStarted","Data":"42c9dc2b659d0106d89989528c3bb8e981438a06085732c073b483c6791a6fbc"}
Sep 17 23:27:26.482613 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:26.482229    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" event={"ID":"a06322e6-cd3d-4665-9b2a-378aba1213dd","Type":"ContainerStarted","Data":"bd58fdc033e4e1b55ca55425ea483dd3ffcf2f0ecf07cf87d50f1ff5638319b3"}
Sep 17 23:27:26.482613 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:26.482521    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:26.482692 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:26.482661    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:26.484110 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:26.484088    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:27:26.503812 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:26.503769    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" podStartSLOduration=2.676622245 podStartE2EDuration="6.503755417s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.827133172s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:26.502662679 +0000 UTC m=+1001.189735256" watchObservedRunningTime="2026-09-17 23:27:26.503755417 +0000 UTC m=+1001.190828079"
Sep 17 23:27:26.870572 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:26.870498    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26_d8f62f64-a4ef-4506-8d68-5803bed81380/kserve-container/0.log"
Sep 17 23:27:27.076324 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.076281    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"]
Sep 17 23:27:27.076540 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.076512    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26" podUID="d8f62f64-a4ef-4506-8d68-5803bed81380" containerName="kserve-container" containerID="cri-o://a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520" gracePeriod=30
Sep 17 23:27:27.076635 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.076561    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26" podUID="d8f62f64-a4ef-4506-8d68-5803bed81380" containerName="kube-rbac-proxy" containerID="cri-o://f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1" gracePeriod=30
Sep 17 23:27:27.203291 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.203262    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"]
Sep 17 23:27:27.206882 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.206860    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.209389 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.209368    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-a284b-predictor-serving-cert\""
Sep 17 23:27:27.209512 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.209420    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-a284b-kube-rbac-proxy-sar-config\""
Sep 17 23:27:27.219483 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.219457    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"]
Sep 17 23:27:27.284568 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.284536    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01e68429-e219-40fa-b661-087ca5136e5e-proxy-tls\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.284568 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.284575    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01e68429-e219-40fa-b661-087ca5136e5e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.284786 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.284662    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd7t9\" (UniqueName: \"kubernetes.io/projected/01e68429-e219-40fa-b661-087ca5136e5e-kube-api-access-xd7t9\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.284786 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.284692    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01e68429-e219-40fa-b661-087ca5136e5e-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.284786 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.284747    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01e68429-e219-40fa-b661-087ca5136e5e-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.327971 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.327948    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:27:27.385851 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.385805    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l857v\" (UniqueName: \"kubernetes.io/projected/d8f62f64-a4ef-4506-8d68-5803bed81380-kube-api-access-l857v\") pod \"d8f62f64-a4ef-4506-8d68-5803bed81380\" (UID: \"d8f62f64-a4ef-4506-8d68-5803bed81380\") "
Sep 17 23:27:27.386048 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.385878    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8f62f64-a4ef-4506-8d68-5803bed81380-kube-rbac-proxy-sar-config\") pod \"d8f62f64-a4ef-4506-8d68-5803bed81380\" (UID: \"d8f62f64-a4ef-4506-8d68-5803bed81380\") "
Sep 17 23:27:27.386048 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.385993    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f62f64-a4ef-4506-8d68-5803bed81380-proxy-tls\") pod \"d8f62f64-a4ef-4506-8d68-5803bed81380\" (UID: \"d8f62f64-a4ef-4506-8d68-5803bed81380\") "
Sep 17 23:27:27.386166 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.386116    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01e68429-e219-40fa-b661-087ca5136e5e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.386221 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.386184    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7t9\" (UniqueName: \"kubernetes.io/projected/01e68429-e219-40fa-b661-087ca5136e5e-kube-api-access-xd7t9\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.386274 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.386223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01e68429-e219-40fa-b661-087ca5136e5e-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.386274 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.386259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01e68429-e219-40fa-b661-087ca5136e5e-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.386386 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.386282    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f62f64-a4ef-4506-8d68-5803bed81380-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d8f62f64-a4ef-4506-8d68-5803bed81380" (UID: "d8f62f64-a4ef-4506-8d68-5803bed81380"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:27.386386 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.386350    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01e68429-e219-40fa-b661-087ca5136e5e-proxy-tls\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.386475 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.386413    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8f62f64-a4ef-4506-8d68-5803bed81380-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:27.386531 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:27:27.386514    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-serving-cert: secret "isvc-sklearn-scale-raw-a284b-predictor-serving-cert" not found
Sep 17 23:27:27.386636 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:27:27.386579    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01e68429-e219-40fa-b661-087ca5136e5e-proxy-tls podName:01e68429-e219-40fa-b661-087ca5136e5e nodeName:}" failed. No retries permitted until 2026-09-17 23:27:27.886555223 +0000 UTC m=+1002.573627784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/01e68429-e219-40fa-b661-087ca5136e5e-proxy-tls") pod "isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" (UID: "01e68429-e219-40fa-b661-087ca5136e5e") : secret "isvc-sklearn-scale-raw-a284b-predictor-serving-cert" not found
Sep 17 23:27:27.386636 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.386603    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01e68429-e219-40fa-b661-087ca5136e5e-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.386930 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.386900    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01e68429-e219-40fa-b661-087ca5136e5e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.387236 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.387209    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01e68429-e219-40fa-b661-087ca5136e5e-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.388432 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.388398    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f62f64-a4ef-4506-8d68-5803bed81380-kube-api-access-l857v" (OuterVolumeSpecName: "kube-api-access-l857v") pod "d8f62f64-a4ef-4506-8d68-5803bed81380" (UID: "d8f62f64-a4ef-4506-8d68-5803bed81380"). InnerVolumeSpecName "kube-api-access-l857v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:27.388432 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.388409    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f62f64-a4ef-4506-8d68-5803bed81380-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d8f62f64-a4ef-4506-8d68-5803bed81380" (UID: "d8f62f64-a4ef-4506-8d68-5803bed81380"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:27:27.397314 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.397291    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7t9\" (UniqueName: \"kubernetes.io/projected/01e68429-e219-40fa-b661-087ca5136e5e-kube-api-access-xd7t9\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.487027 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.486942    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l857v\" (UniqueName: \"kubernetes.io/projected/d8f62f64-a4ef-4506-8d68-5803bed81380-kube-api-access-l857v\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:27.487027 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.486970    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8f62f64-a4ef-4506-8d68-5803bed81380-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:27:27.487610 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.487581    2582 generic.go:356] "Generic (PLEG): container finished" podID="d8f62f64-a4ef-4506-8d68-5803bed81380" containerID="f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1" exitCode=2
Sep 17 23:27:27.487716 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.487611    2582 generic.go:356] "Generic (PLEG): container finished" podID="d8f62f64-a4ef-4506-8d68-5803bed81380" containerID="a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520" exitCode=2
Sep 17 23:27:27.487716 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.487635    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"
Sep 17 23:27:27.487716 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.487651    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26" event={"ID":"d8f62f64-a4ef-4506-8d68-5803bed81380","Type":"ContainerDied","Data":"f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1"}
Sep 17 23:27:27.487865 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.487717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26" event={"ID":"d8f62f64-a4ef-4506-8d68-5803bed81380","Type":"ContainerDied","Data":"a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520"}
Sep 17 23:27:27.487865 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.487748    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26" event={"ID":"d8f62f64-a4ef-4506-8d68-5803bed81380","Type":"ContainerDied","Data":"28ca5aff3db8dd6aa4036e172a0903fd41ea3ca5515e17e536c00dc125f7e3a0"}
Sep 17 23:27:27.487865 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.487784    2582 scope.go:117] "RemoveContainer" containerID="f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1"
Sep 17 23:27:27.488369 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.488340    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:27:27.496090 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.496071    2582 scope.go:117] "RemoveContainer" containerID="a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520"
Sep 17 23:27:27.503280 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.503265    2582 scope.go:117] "RemoveContainer" containerID="f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1"
Sep 17 23:27:27.503551 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:27:27.503531    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1\": container with ID starting with f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1 not found: ID does not exist" containerID="f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1"
Sep 17 23:27:27.503601 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.503561    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1"} err="failed to get container status \"f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1\": rpc error: code = NotFound desc = could not find container \"f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1\": container with ID starting with f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1 not found: ID does not exist"
Sep 17 23:27:27.503601 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.503578    2582 scope.go:117] "RemoveContainer" containerID="a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520"
Sep 17 23:27:27.503861 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:27:27.503822    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520\": container with ID starting with a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520 not found: ID does not exist" containerID="a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520"
Sep 17 23:27:27.503976 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.503870    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520"} err="failed to get container status \"a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520\": rpc error: code = NotFound desc = could not find container \"a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520\": container with ID starting with a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520 not found: ID does not exist"
Sep 17 23:27:27.503976 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.503893    2582 scope.go:117] "RemoveContainer" containerID="f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1"
Sep 17 23:27:27.504148 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.504129    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1"} err="failed to get container status \"f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1\": rpc error: code = NotFound desc = could not find container \"f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1\": container with ID starting with f1e4f4667a3420fa7f2dd253b187b350225e86b6d0b79adbf5aa9e1e74e910c1 not found: ID does not exist"
Sep 17 23:27:27.504217 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.504150    2582 scope.go:117] "RemoveContainer" containerID="a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520"
Sep 17 23:27:27.504398 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.504379    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520"} err="failed to get container status \"a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520\": rpc error: code = NotFound desc = could not find container \"a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520\": container with ID starting with a41aba551ca1022a0053da21fe48a99dd9105272f483261b86d0d88ad31bb520 not found: ID does not exist"
Sep 17 23:27:27.508862 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.508822    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"]
Sep 17 23:27:27.510914 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.510896    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-0a0da-predictor-c8fbfbc9d-krl26"]
Sep 17 23:27:27.890781 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.890750    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01e68429-e219-40fa-b661-087ca5136e5e-proxy-tls\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:27.893345 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:27.893327    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01e68429-e219-40fa-b661-087ca5136e5e-proxy-tls\") pod \"isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:28.119279 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:28.119234    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:28.397081 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:28.397058    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"]
Sep 17 23:27:28.399237 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:27:28.399204    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e68429_e219_40fa_b661_087ca5136e5e.slice/crio-0207c8d963b4f268a248271179c519cdf312f3bdb7ce488c3ba6dcf964596a1d WatchSource:0}: Error finding container 0207c8d963b4f268a248271179c519cdf312f3bdb7ce488c3ba6dcf964596a1d: Status 404 returned error can't find the container with id 0207c8d963b4f268a248271179c519cdf312f3bdb7ce488c3ba6dcf964596a1d
Sep 17 23:27:28.494621 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:28.494584    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" event={"ID":"01e68429-e219-40fa-b661-087ca5136e5e","Type":"ContainerStarted","Data":"368f0a4d6fc5239def1e039b2efdfa8d1fdb357416e87b671bf9443a6eb50247"}
Sep 17 23:27:28.495068 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:28.494642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" event={"ID":"01e68429-e219-40fa-b661-087ca5136e5e","Type":"ContainerStarted","Data":"0207c8d963b4f268a248271179c519cdf312f3bdb7ce488c3ba6dcf964596a1d"}
Sep 17 23:27:29.393616 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:29.393579    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8f62f64-a4ef-4506-8d68-5803bed81380" path="/var/lib/kubelet/pods/d8f62f64-a4ef-4506-8d68-5803bed81380/volumes"
Sep 17 23:27:32.492890 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:32.492857    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:27:32.493441 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:32.493415    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:27:32.513341 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:32.513308    2582 generic.go:356] "Generic (PLEG): container finished" podID="01e68429-e219-40fa-b661-087ca5136e5e" containerID="368f0a4d6fc5239def1e039b2efdfa8d1fdb357416e87b671bf9443a6eb50247" exitCode=0
Sep 17 23:27:32.513516 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:32.513383    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" event={"ID":"01e68429-e219-40fa-b661-087ca5136e5e","Type":"ContainerDied","Data":"368f0a4d6fc5239def1e039b2efdfa8d1fdb357416e87b671bf9443a6eb50247"}
Sep 17 23:27:33.519824 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:33.519792    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" event={"ID":"01e68429-e219-40fa-b661-087ca5136e5e","Type":"ContainerStarted","Data":"c8d7450b3226079b35c4c0173b259f51c27f908949ba907cb2392e9fab4608dc"}
Sep 17 23:27:33.520241 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:33.519874    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" event={"ID":"01e68429-e219-40fa-b661-087ca5136e5e","Type":"ContainerStarted","Data":"1c9498c1289f40dc526dab79da4c6bb4ec1727014512488207182b3a8be04ebb"}
Sep 17 23:27:33.520241 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:33.520059    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:33.540825 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:33.540771    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podStartSLOduration=2.68415189 podStartE2EDuration="6.540756041s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.856604151s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:33.540042571 +0000 UTC m=+1008.227115147" watchObservedRunningTime="2026-09-17 23:27:33.540756041 +0000 UTC m=+1008.227828618"
Sep 17 23:27:34.524412 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:34.524380    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:34.525690 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:34.525659    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:27:35.528592 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:35.528545    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:27:40.533889 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:40.533857    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:27:40.534486 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:40.534455    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:27:42.493691 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:42.493651    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:27:50.534940 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:50.534897    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:27:52.493491 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:27:52.493440    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:28:00.534434 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:00.534388    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:28:02.494139 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:02.494099    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:28:10.535232 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:10.535193    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:28:12.493724 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:12.493682    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:28:20.534476 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:20.534437    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:28:22.494237 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:22.494192    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:28:30.534566 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:30.534475    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:28:32.493486 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:32.493450    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:28:40.534655 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:40.534613    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:28:41.393534 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:41.393499    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:28:50.535029 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:50.534984    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:28:50.592176 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:50.592135    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"]
Sep 17 23:28:50.592416 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:50.592386    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kserve-container" containerID="cri-o://bd58fdc033e4e1b55ca55425ea483dd3ffcf2f0ecf07cf87d50f1ff5638319b3" gracePeriod=30
Sep 17 23:28:50.592559 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:50.592420    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kube-rbac-proxy" containerID="cri-o://42c9dc2b659d0106d89989528c3bb8e981438a06085732c073b483c6791a6fbc" gracePeriod=30
Sep 17 23:28:50.832602 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:50.832502    2582 generic.go:356] "Generic (PLEG): container finished" podID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerID="42c9dc2b659d0106d89989528c3bb8e981438a06085732c073b483c6791a6fbc" exitCode=2
Sep 17 23:28:50.832602 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:50.832570    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" event={"ID":"a06322e6-cd3d-4665-9b2a-378aba1213dd","Type":"ContainerDied","Data":"42c9dc2b659d0106d89989528c3bb8e981438a06085732c073b483c6791a6fbc"}
Sep 17 23:28:51.389594 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:51.389550    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 23:28:52.488755 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:52.488718    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 23:28:55.547946 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.547915    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:28:55.606313 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.606284    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a06322e6-cd3d-4665-9b2a-378aba1213dd-kube-rbac-proxy-sar-config\") pod \"a06322e6-cd3d-4665-9b2a-378aba1213dd\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") "
Sep 17 23:28:55.606525 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.606326    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htd6m\" (UniqueName: \"kubernetes.io/projected/a06322e6-cd3d-4665-9b2a-378aba1213dd-kube-api-access-htd6m\") pod \"a06322e6-cd3d-4665-9b2a-378aba1213dd\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") "
Sep 17 23:28:55.606525 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.606355    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a06322e6-cd3d-4665-9b2a-378aba1213dd-cabundle-cert\") pod \"a06322e6-cd3d-4665-9b2a-378aba1213dd\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") "
Sep 17 23:28:55.606525 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.606418    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a06322e6-cd3d-4665-9b2a-378aba1213dd-proxy-tls\") pod \"a06322e6-cd3d-4665-9b2a-378aba1213dd\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") "
Sep 17 23:28:55.606525 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.606458    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a06322e6-cd3d-4665-9b2a-378aba1213dd-kserve-provision-location\") pod \"a06322e6-cd3d-4665-9b2a-378aba1213dd\" (UID: \"a06322e6-cd3d-4665-9b2a-378aba1213dd\") "
Sep 17 23:28:55.606732 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.606642    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06322e6-cd3d-4665-9b2a-378aba1213dd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a06322e6-cd3d-4665-9b2a-378aba1213dd" (UID: "a06322e6-cd3d-4665-9b2a-378aba1213dd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:28:55.606789 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.606728    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a06322e6-cd3d-4665-9b2a-378aba1213dd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:28:55.606962 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.606937    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06322e6-cd3d-4665-9b2a-378aba1213dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a06322e6-cd3d-4665-9b2a-378aba1213dd" (UID: "a06322e6-cd3d-4665-9b2a-378aba1213dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:28:55.607052 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.606974    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06322e6-cd3d-4665-9b2a-378aba1213dd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a06322e6-cd3d-4665-9b2a-378aba1213dd" (UID: "a06322e6-cd3d-4665-9b2a-378aba1213dd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:28:55.609486 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.609447    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06322e6-cd3d-4665-9b2a-378aba1213dd-kube-api-access-htd6m" (OuterVolumeSpecName: "kube-api-access-htd6m") pod "a06322e6-cd3d-4665-9b2a-378aba1213dd" (UID: "a06322e6-cd3d-4665-9b2a-378aba1213dd"). InnerVolumeSpecName "kube-api-access-htd6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:28:55.609692 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.609675    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a06322e6-cd3d-4665-9b2a-378aba1213dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a06322e6-cd3d-4665-9b2a-378aba1213dd" (UID: "a06322e6-cd3d-4665-9b2a-378aba1213dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:28:55.695859 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.695802    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"]
Sep 17 23:28:55.696120 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.696107    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="storage-initializer"
Sep 17 23:28:55.696190 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.696122    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="storage-initializer"
Sep 17 23:28:55.696190 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.696132    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kserve-container"
Sep 17 23:28:55.696190 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.696138    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kserve-container"
Sep 17 23:28:55.696190 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.696148    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8f62f64-a4ef-4506-8d68-5803bed81380" containerName="kube-rbac-proxy"
Sep 17 23:28:55.696190 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.696154    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8f62f64-a4ef-4506-8d68-5803bed81380" containerName="kube-rbac-proxy"
Sep 17 23:28:55.696190 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.696160    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kube-rbac-proxy"
Sep 17 23:28:55.696190 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.696165    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kube-rbac-proxy"
Sep 17 23:28:55.696190 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.696178    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8f62f64-a4ef-4506-8d68-5803bed81380" containerName="kserve-container"
Sep 17 23:28:55.696190 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.696183    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8f62f64-a4ef-4506-8d68-5803bed81380" containerName="kserve-container"
Sep 17 23:28:55.696626 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.696231    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8f62f64-a4ef-4506-8d68-5803bed81380" containerName="kserve-container"
Sep 17 23:28:55.696626 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.696240    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8f62f64-a4ef-4506-8d68-5803bed81380" containerName="kube-rbac-proxy"
Sep 17 23:28:55.696626 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.696248    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kube-rbac-proxy"
Sep 17 23:28:55.696626 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.696254    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerName="kserve-container"
Sep 17 23:28:55.699644 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.699627    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:55.703226 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.703196    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-60b0d-predictor-serving-cert\""
Sep 17 23:28:55.703370 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.703245    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-60b0d-kube-rbac-proxy-sar-config\""
Sep 17 23:28:55.707089 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.707072    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a06322e6-cd3d-4665-9b2a-378aba1213dd-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:28:55.707180 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.707092    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a06322e6-cd3d-4665-9b2a-378aba1213dd-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:28:55.707180 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.707108    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htd6m\" (UniqueName: \"kubernetes.io/projected/a06322e6-cd3d-4665-9b2a-378aba1213dd-kube-api-access-htd6m\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:28:55.707180 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.707119    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a06322e6-cd3d-4665-9b2a-378aba1213dd-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:28:55.709758 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.709728    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"]
Sep 17 23:28:55.808281 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.808238    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03882ad6-d499-41f5-aece-c3c5ea157867-proxy-tls\") pod \"raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:55.808493 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.808297    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03882ad6-d499-41f5-aece-c3c5ea157867-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:55.808493 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.808329    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03882ad6-d499-41f5-aece-c3c5ea157867-kserve-provision-location\") pod \"raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:55.808493 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.808382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03882ad6-d499-41f5-aece-c3c5ea157867-cabundle-cert\") pod \"raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:55.808493 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.808411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrv7x\" (UniqueName: \"kubernetes.io/projected/03882ad6-d499-41f5-aece-c3c5ea157867-kube-api-access-hrv7x\") pod \"raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:55.854539 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.854504    2582 generic.go:356] "Generic (PLEG): container finished" podID="a06322e6-cd3d-4665-9b2a-378aba1213dd" containerID="bd58fdc033e4e1b55ca55425ea483dd3ffcf2f0ecf07cf87d50f1ff5638319b3" exitCode=0
Sep 17 23:28:55.854692 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.854555    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" event={"ID":"a06322e6-cd3d-4665-9b2a-378aba1213dd","Type":"ContainerDied","Data":"bd58fdc033e4e1b55ca55425ea483dd3ffcf2f0ecf07cf87d50f1ff5638319b3"}
Sep 17 23:28:55.854692 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.854574    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"
Sep 17 23:28:55.854692 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.854605    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r" event={"ID":"a06322e6-cd3d-4665-9b2a-378aba1213dd","Type":"ContainerDied","Data":"c943058d1d6028a6f86073b6fb4e73835a2e67aa8ece1ba0c4d5c60db6e7e9e6"}
Sep 17 23:28:55.854692 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.854632    2582 scope.go:117] "RemoveContainer" containerID="42c9dc2b659d0106d89989528c3bb8e981438a06085732c073b483c6791a6fbc"
Sep 17 23:28:55.863685 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.863665    2582 scope.go:117] "RemoveContainer" containerID="bd58fdc033e4e1b55ca55425ea483dd3ffcf2f0ecf07cf87d50f1ff5638319b3"
Sep 17 23:28:55.872560 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.872540    2582 scope.go:117] "RemoveContainer" containerID="eee04f85479105907b821a142ab7ff312e5c72a5e33e52c69ca5aeb2d9f353e2"
Sep 17 23:28:55.878037 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.878010    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"]
Sep 17 23:28:55.880664 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.880647    2582 scope.go:117] "RemoveContainer" containerID="42c9dc2b659d0106d89989528c3bb8e981438a06085732c073b483c6791a6fbc"
Sep 17 23:28:55.881034 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:28:55.881012    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c9dc2b659d0106d89989528c3bb8e981438a06085732c073b483c6791a6fbc\": container with ID starting with 42c9dc2b659d0106d89989528c3bb8e981438a06085732c073b483c6791a6fbc not found: ID does not exist" containerID="42c9dc2b659d0106d89989528c3bb8e981438a06085732c073b483c6791a6fbc"
Sep 17 23:28:55.881131 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.881045    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c9dc2b659d0106d89989528c3bb8e981438a06085732c073b483c6791a6fbc"} err="failed to get container status \"42c9dc2b659d0106d89989528c3bb8e981438a06085732c073b483c6791a6fbc\": rpc error: code = NotFound desc = could not find container \"42c9dc2b659d0106d89989528c3bb8e981438a06085732c073b483c6791a6fbc\": container with ID starting with 42c9dc2b659d0106d89989528c3bb8e981438a06085732c073b483c6791a6fbc not found: ID does not exist"
Sep 17 23:28:55.881131 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.881070    2582 scope.go:117] "RemoveContainer" containerID="bd58fdc033e4e1b55ca55425ea483dd3ffcf2f0ecf07cf87d50f1ff5638319b3"
Sep 17 23:28:55.881224 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.881070    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-35411-predictor-65bc89658c-dth4r"]
Sep 17 23:28:55.881363 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:28:55.881347    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd58fdc033e4e1b55ca55425ea483dd3ffcf2f0ecf07cf87d50f1ff5638319b3\": container with ID starting with bd58fdc033e4e1b55ca55425ea483dd3ffcf2f0ecf07cf87d50f1ff5638319b3 not found: ID does not exist" containerID="bd58fdc033e4e1b55ca55425ea483dd3ffcf2f0ecf07cf87d50f1ff5638319b3"
Sep 17 23:28:55.881430 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.881370    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd58fdc033e4e1b55ca55425ea483dd3ffcf2f0ecf07cf87d50f1ff5638319b3"} err="failed to get container status \"bd58fdc033e4e1b55ca55425ea483dd3ffcf2f0ecf07cf87d50f1ff5638319b3\": rpc error: code = NotFound desc = could not find container \"bd58fdc033e4e1b55ca55425ea483dd3ffcf2f0ecf07cf87d50f1ff5638319b3\": container with ID starting with bd58fdc033e4e1b55ca55425ea483dd3ffcf2f0ecf07cf87d50f1ff5638319b3 not found: ID does not exist"
Sep 17 23:28:55.881430 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.881388    2582 scope.go:117] "RemoveContainer" containerID="eee04f85479105907b821a142ab7ff312e5c72a5e33e52c69ca5aeb2d9f353e2"
Sep 17 23:28:55.881661 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:28:55.881643    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee04f85479105907b821a142ab7ff312e5c72a5e33e52c69ca5aeb2d9f353e2\": container with ID starting with eee04f85479105907b821a142ab7ff312e5c72a5e33e52c69ca5aeb2d9f353e2 not found: ID does not exist" containerID="eee04f85479105907b821a142ab7ff312e5c72a5e33e52c69ca5aeb2d9f353e2"
Sep 17 23:28:55.881703 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.881665    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee04f85479105907b821a142ab7ff312e5c72a5e33e52c69ca5aeb2d9f353e2"} err="failed to get container status \"eee04f85479105907b821a142ab7ff312e5c72a5e33e52c69ca5aeb2d9f353e2\": rpc error: code = NotFound desc = could not find container \"eee04f85479105907b821a142ab7ff312e5c72a5e33e52c69ca5aeb2d9f353e2\": container with ID starting with eee04f85479105907b821a142ab7ff312e5c72a5e33e52c69ca5aeb2d9f353e2 not found: ID does not exist"
Sep 17 23:28:55.909202 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.909158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03882ad6-d499-41f5-aece-c3c5ea157867-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:55.909202 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.909203    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03882ad6-d499-41f5-aece-c3c5ea157867-kserve-provision-location\") pod \"raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:55.909420 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.909252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03882ad6-d499-41f5-aece-c3c5ea157867-cabundle-cert\") pod \"raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:55.909420 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.909281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrv7x\" (UniqueName: \"kubernetes.io/projected/03882ad6-d499-41f5-aece-c3c5ea157867-kube-api-access-hrv7x\") pod \"raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:55.909518 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.909417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03882ad6-d499-41f5-aece-c3c5ea157867-proxy-tls\") pod \"raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:55.909673 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.909656    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03882ad6-d499-41f5-aece-c3c5ea157867-kserve-provision-location\") pod \"raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:55.910003 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.909984    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03882ad6-d499-41f5-aece-c3c5ea157867-cabundle-cert\") pod \"raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:55.910083 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.909983    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03882ad6-d499-41f5-aece-c3c5ea157867-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:55.912043 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.912024    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03882ad6-d499-41f5-aece-c3c5ea157867-proxy-tls\") pod \"raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:55.918119 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:55.918099    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrv7x\" (UniqueName: \"kubernetes.io/projected/03882ad6-d499-41f5-aece-c3c5ea157867-kube-api-access-hrv7x\") pod \"raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:56.010101 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:56.010004    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:28:56.283737 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:56.283711    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"]
Sep 17 23:28:56.285785 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:28:56.285754    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03882ad6_d499_41f5_aece_c3c5ea157867.slice/crio-0b4a03751bd585f1ea917fe8239f693a9d6fadb9a55128dfd7e489ebae4a0afd WatchSource:0}: Error finding container 0b4a03751bd585f1ea917fe8239f693a9d6fadb9a55128dfd7e489ebae4a0afd: Status 404 returned error can't find the container with id 0b4a03751bd585f1ea917fe8239f693a9d6fadb9a55128dfd7e489ebae4a0afd
Sep 17 23:28:56.861375 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:56.861338    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" event={"ID":"03882ad6-d499-41f5-aece-c3c5ea157867","Type":"ContainerStarted","Data":"e000888896a0bc102f22fda0d92c879e4f06cdd9837985d0a2794d06835328df"}
Sep 17 23:28:56.861763 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:56.861430    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" event={"ID":"03882ad6-d499-41f5-aece-c3c5ea157867","Type":"ContainerStarted","Data":"0b4a03751bd585f1ea917fe8239f693a9d6fadb9a55128dfd7e489ebae4a0afd"}
Sep 17 23:28:57.393885 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:28:57.393820    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a06322e6-cd3d-4665-9b2a-378aba1213dd" path="/var/lib/kubelet/pods/a06322e6-cd3d-4665-9b2a-378aba1213dd/volumes"
Sep 17 23:29:00.534604 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:00.534561    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:29:00.879290 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:00.879201    2582 generic.go:356] "Generic (PLEG): container finished" podID="03882ad6-d499-41f5-aece-c3c5ea157867" containerID="e000888896a0bc102f22fda0d92c879e4f06cdd9837985d0a2794d06835328df" exitCode=0
Sep 17 23:29:00.879290 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:00.879275    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" event={"ID":"03882ad6-d499-41f5-aece-c3c5ea157867","Type":"ContainerDied","Data":"e000888896a0bc102f22fda0d92c879e4f06cdd9837985d0a2794d06835328df"}
Sep 17 23:29:01.885757 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:01.885719    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" event={"ID":"03882ad6-d499-41f5-aece-c3c5ea157867","Type":"ContainerStarted","Data":"ede4de5eb70b7370e978d708b9948ac994b6a00bc48e15a0a43b34f1eda5c218"}
Sep 17 23:29:01.886393 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:01.885786    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" event={"ID":"03882ad6-d499-41f5-aece-c3c5ea157867","Type":"ContainerStarted","Data":"46d67d0e31c1dcf856c79dee500b1ff806e3e7c2a893c5cb230ffa77e0b5a3f7"}
Sep 17 23:29:01.886393 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:01.886090    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:29:01.886393 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:01.886267    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:29:01.887738 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:01.887714    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:29:01.909985 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:01.909925    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" podStartSLOduration=2.83239806 podStartE2EDuration="6.909908927s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.077510867s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:29:01.909421011 +0000 UTC m=+1096.596493590" watchObservedRunningTime="2026-09-17 23:29:01.909908927 +0000 UTC m=+1096.596981504"
Sep 17 23:29:02.889993 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:02.889955    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:29:07.894782 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:07.894748    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:29:07.895286 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:07.895260    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:29:10.389607 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:10.389565    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:29:17.895286 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:17.895243    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:29:20.389695 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:20.389644    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:29:27.896258 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:27.896218    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:29:30.389901 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:30.389823    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:29:37.895199 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:37.895150    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:29:40.389820 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:40.389780    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:29:47.895986 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:47.895947    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:29:50.389622 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:50.389575    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:29:57.895415 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:29:57.895322    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:30:00.390183 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:00.390131    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:30:07.895269 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:07.895225    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:30:10.390950 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:10.390920    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:30:17.257648 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:17.257615    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"]
Sep 17 23:30:17.258165 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:17.257913    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" containerID="cri-o://1c9498c1289f40dc526dab79da4c6bb4ec1727014512488207182b3a8be04ebb" gracePeriod=30
Sep 17 23:30:17.258165 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:17.257945    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kube-rbac-proxy" containerID="cri-o://c8d7450b3226079b35c4c0173b259f51c27f908949ba907cb2392e9fab4608dc" gracePeriod=30
Sep 17 23:30:17.895906 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:17.895876    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:30:18.180899 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:18.180798    2582 generic.go:356] "Generic (PLEG): container finished" podID="01e68429-e219-40fa-b661-087ca5136e5e" containerID="c8d7450b3226079b35c4c0173b259f51c27f908949ba907cb2392e9fab4608dc" exitCode=2
Sep 17 23:30:18.180899 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:18.180871    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" event={"ID":"01e68429-e219-40fa-b661-087ca5136e5e","Type":"ContainerDied","Data":"c8d7450b3226079b35c4c0173b259f51c27f908949ba907cb2392e9fab4608dc"}
Sep 17 23:30:20.390226 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:20.390185    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 23:30:20.529028 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:20.528985    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 17 23:30:22.498345 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.498311    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"]
Sep 17 23:30:22.501445 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.501426    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:22.504920 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.504896    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 23:30:22.505331 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.505309    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 23:30:22.524536 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.524503    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"]
Sep 17 23:30:22.600635 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.600593    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"]
Sep 17 23:30:22.604766 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.604743    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:22.605364 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.605336    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:22.605470 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.605387    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f4f16b7-e915-4ec5-8540-c4c38e713afe-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:22.605515 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.605500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:22.605551 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.605533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h88l2\" (UniqueName: \"kubernetes.io/projected/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kube-api-access-h88l2\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:22.605591 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.605566    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f4f16b7-e915-4ec5-8540-c4c38e713afe-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:22.607498 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.607477    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 23:30:22.615300 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.615271    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"]
Sep 17 23:30:22.706672 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.706640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:22.706672 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.706676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h88l2\" (UniqueName: \"kubernetes.io/projected/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kube-api-access-h88l2\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:22.706962 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.706698    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65169dc-7302-4d5f-8fd1-a09b169f5229-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:22.706962 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.706717    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f4f16b7-e915-4ec5-8540-c4c38e713afe-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:22.706962 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.706751    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:22.706962 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.706805    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65169dc-7302-4d5f-8fd1-a09b169f5229-proxy-tls\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:22.706962 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.706897    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f4f16b7-e915-4ec5-8540-c4c38e713afe-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:22.706962 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.706928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65169dc-7302-4d5f-8fd1-a09b169f5229-kserve-provision-location\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:22.706962 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.706962    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgrd2\" (UniqueName: \"kubernetes.io/projected/a65169dc-7302-4d5f-8fd1-a09b169f5229-kube-api-access-xgrd2\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:22.707280 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.707004    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65169dc-7302-4d5f-8fd1-a09b169f5229-cabundle-cert\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:22.707280 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:30:22.707177    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 23:30:22.707280 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:30:22.707240    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f4f16b7-e915-4ec5-8540-c4c38e713afe-proxy-tls podName:5f4f16b7-e915-4ec5-8540-c4c38e713afe nodeName:}" failed. No retries permitted until 2026-09-17 23:30:23.20722009 +0000 UTC m=+1177.894292652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5f4f16b7-e915-4ec5-8540-c4c38e713afe-proxy-tls") pod "isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" (UID: "5f4f16b7-e915-4ec5-8540-c4c38e713afe") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 17 23:30:22.707280 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.707243    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:22.707485 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.707467    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f4f16b7-e915-4ec5-8540-c4c38e713afe-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:22.707544 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.707525    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:22.715683 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.715659    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h88l2\" (UniqueName: \"kubernetes.io/projected/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kube-api-access-h88l2\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:22.808370 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.808267    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65169dc-7302-4d5f-8fd1-a09b169f5229-kserve-provision-location\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:22.808370 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.808315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgrd2\" (UniqueName: \"kubernetes.io/projected/a65169dc-7302-4d5f-8fd1-a09b169f5229-kube-api-access-xgrd2\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:22.808654 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.808456    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65169dc-7302-4d5f-8fd1-a09b169f5229-cabundle-cert\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:22.808654 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.808559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65169dc-7302-4d5f-8fd1-a09b169f5229-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:22.808771 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.808661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65169dc-7302-4d5f-8fd1-a09b169f5229-proxy-tls\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:22.808771 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.808741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65169dc-7302-4d5f-8fd1-a09b169f5229-kserve-provision-location\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:22.808893 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:30:22.808795    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 23:30:22.808893 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:30:22.808875    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65169dc-7302-4d5f-8fd1-a09b169f5229-proxy-tls podName:a65169dc-7302-4d5f-8fd1-a09b169f5229 nodeName:}" failed. No retries permitted until 2026-09-17 23:30:23.308854587 +0000 UTC m=+1177.995927147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a65169dc-7302-4d5f-8fd1-a09b169f5229-proxy-tls") pod "isvc-canary-create-predictor-65f7bb5954-5hl44" (UID: "a65169dc-7302-4d5f-8fd1-a09b169f5229") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 23:30:22.809195 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.809176    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65169dc-7302-4d5f-8fd1-a09b169f5229-cabundle-cert\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:22.809323 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.809303    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65169dc-7302-4d5f-8fd1-a09b169f5229-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:22.818234 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:22.818210    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgrd2\" (UniqueName: \"kubernetes.io/projected/a65169dc-7302-4d5f-8fd1-a09b169f5229-kube-api-access-xgrd2\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:23.212383 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:23.212348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f4f16b7-e915-4ec5-8540-c4c38e713afe-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:23.215011 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:23.214985    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f4f16b7-e915-4ec5-8540-c4c38e713afe-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:23.312935 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:23.312881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65169dc-7302-4d5f-8fd1-a09b169f5229-proxy-tls\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:23.315540 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:23.315517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65169dc-7302-4d5f-8fd1-a09b169f5229-proxy-tls\") pod \"isvc-canary-create-predictor-65f7bb5954-5hl44\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:23.411798 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:23.411767    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:23.516314 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:23.516271    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:23.688611 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:23.688578    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"]
Sep 17 23:30:23.690540 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:30:23.690512    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4f16b7_e915_4ec5_8540_c4c38e713afe.slice/crio-d9452d9a03c9c4e1d633ba82506ecb418db1ffa93b9ce593e0b3aaaae0ca72e8 WatchSource:0}: Error finding container d9452d9a03c9c4e1d633ba82506ecb418db1ffa93b9ce593e0b3aaaae0ca72e8: Status 404 returned error can't find the container with id d9452d9a03c9c4e1d633ba82506ecb418db1ffa93b9ce593e0b3aaaae0ca72e8
Sep 17 23:30:23.789531 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:23.789500    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"]
Sep 17 23:30:23.791388 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:30:23.791355    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65169dc_7302_4d5f_8fd1_a09b169f5229.slice/crio-f51db9769753e1d525f8e9b0eca5e0fae29fa8361c3bd636ffed71613cb73dfd WatchSource:0}: Error finding container f51db9769753e1d525f8e9b0eca5e0fae29fa8361c3bd636ffed71613cb73dfd: Status 404 returned error can't find the container with id f51db9769753e1d525f8e9b0eca5e0fae29fa8361c3bd636ffed71613cb73dfd
Sep 17 23:30:24.205303 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:24.205234    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" event={"ID":"a65169dc-7302-4d5f-8fd1-a09b169f5229","Type":"ContainerStarted","Data":"fb9525c8c49e4e068c9acc0023744917956a8c8a65de4d67c83ca1f89208546a"}
Sep 17 23:30:24.205525 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:24.205312    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" event={"ID":"a65169dc-7302-4d5f-8fd1-a09b169f5229","Type":"ContainerStarted","Data":"f51db9769753e1d525f8e9b0eca5e0fae29fa8361c3bd636ffed71613cb73dfd"}
Sep 17 23:30:24.207040 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:24.207008    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" event={"ID":"5f4f16b7-e915-4ec5-8540-c4c38e713afe","Type":"ContainerStarted","Data":"91cdc5f4e684804217d67e0a7343e18faa0f698478dad0fdf370120287c70376"}
Sep 17 23:30:24.207133 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:24.207085    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" event={"ID":"5f4f16b7-e915-4ec5-8540-c4c38e713afe","Type":"ContainerStarted","Data":"d9452d9a03c9c4e1d633ba82506ecb418db1ffa93b9ce593e0b3aaaae0ca72e8"}
Sep 17 23:30:25.529245 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:25.529200    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 17 23:30:25.826737 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:25.826650    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"]
Sep 17 23:30:25.826936 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:25.826900    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kserve-container" containerID="cri-o://46d67d0e31c1dcf856c79dee500b1ff806e3e7c2a893c5cb230ffa77e0b5a3f7" gracePeriod=30
Sep 17 23:30:25.827039 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:25.826990    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kube-rbac-proxy" containerID="cri-o://ede4de5eb70b7370e978d708b9948ac994b6a00bc48e15a0a43b34f1eda5c218" gracePeriod=30
Sep 17 23:30:26.217068 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:26.217031    2582 generic.go:356] "Generic (PLEG): container finished" podID="03882ad6-d499-41f5-aece-c3c5ea157867" containerID="ede4de5eb70b7370e978d708b9948ac994b6a00bc48e15a0a43b34f1eda5c218" exitCode=2
Sep 17 23:30:26.217245 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:26.217088    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" event={"ID":"03882ad6-d499-41f5-aece-c3c5ea157867","Type":"ContainerDied","Data":"ede4de5eb70b7370e978d708b9948ac994b6a00bc48e15a0a43b34f1eda5c218"}
Sep 17 23:30:27.810248 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:27.810223    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:30:27.890714 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:27.890670    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 23:30:27.895998 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:27.895969    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 23:30:27.954770 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:27.954733    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01e68429-e219-40fa-b661-087ca5136e5e-proxy-tls\") pod \"01e68429-e219-40fa-b661-087ca5136e5e\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") "
Sep 17 23:30:27.954770 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:27.954775    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd7t9\" (UniqueName: \"kubernetes.io/projected/01e68429-e219-40fa-b661-087ca5136e5e-kube-api-access-xd7t9\") pod \"01e68429-e219-40fa-b661-087ca5136e5e\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") "
Sep 17 23:30:27.954996 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:27.954797    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01e68429-e219-40fa-b661-087ca5136e5e-kube-rbac-proxy-sar-config\") pod \"01e68429-e219-40fa-b661-087ca5136e5e\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") "
Sep 17 23:30:27.954996 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:27.954818    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01e68429-e219-40fa-b661-087ca5136e5e-cabundle-cert\") pod \"01e68429-e219-40fa-b661-087ca5136e5e\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") "
Sep 17 23:30:27.954996 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:27.954870    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01e68429-e219-40fa-b661-087ca5136e5e-kserve-provision-location\") pod \"01e68429-e219-40fa-b661-087ca5136e5e\" (UID: \"01e68429-e219-40fa-b661-087ca5136e5e\") "
Sep 17 23:30:27.955252 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:27.955230    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e68429-e219-40fa-b661-087ca5136e5e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "01e68429-e219-40fa-b661-087ca5136e5e" (UID: "01e68429-e219-40fa-b661-087ca5136e5e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:27.955317 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:27.955236    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e68429-e219-40fa-b661-087ca5136e5e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "01e68429-e219-40fa-b661-087ca5136e5e" (UID: "01e68429-e219-40fa-b661-087ca5136e5e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:30:27.955317 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:27.955247    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e68429-e219-40fa-b661-087ca5136e5e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "01e68429-e219-40fa-b661-087ca5136e5e" (UID: "01e68429-e219-40fa-b661-087ca5136e5e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:30:27.957254 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:27.957231    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e68429-e219-40fa-b661-087ca5136e5e-kube-api-access-xd7t9" (OuterVolumeSpecName: "kube-api-access-xd7t9") pod "01e68429-e219-40fa-b661-087ca5136e5e" (UID: "01e68429-e219-40fa-b661-087ca5136e5e"). InnerVolumeSpecName "kube-api-access-xd7t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:30:27.957320 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:27.957283    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e68429-e219-40fa-b661-087ca5136e5e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "01e68429-e219-40fa-b661-087ca5136e5e" (UID: "01e68429-e219-40fa-b661-087ca5136e5e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:30:28.056363 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.056276    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01e68429-e219-40fa-b661-087ca5136e5e-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:30:28.056363 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.056308    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd7t9\" (UniqueName: \"kubernetes.io/projected/01e68429-e219-40fa-b661-087ca5136e5e-kube-api-access-xd7t9\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:30:28.056363 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.056318    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01e68429-e219-40fa-b661-087ca5136e5e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:30:28.056363 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.056328    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01e68429-e219-40fa-b661-087ca5136e5e-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:30:28.056363 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.056337    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01e68429-e219-40fa-b661-087ca5136e5e-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:30:28.227671 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.227637    2582 generic.go:356] "Generic (PLEG): container finished" podID="01e68429-e219-40fa-b661-087ca5136e5e" containerID="1c9498c1289f40dc526dab79da4c6bb4ec1727014512488207182b3a8be04ebb" exitCode=0
Sep 17 23:30:28.227877 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.227697    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"
Sep 17 23:30:28.227877 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.227710    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" event={"ID":"01e68429-e219-40fa-b661-087ca5136e5e","Type":"ContainerDied","Data":"1c9498c1289f40dc526dab79da4c6bb4ec1727014512488207182b3a8be04ebb"}
Sep 17 23:30:28.227877 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.227768    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l" event={"ID":"01e68429-e219-40fa-b661-087ca5136e5e","Type":"ContainerDied","Data":"0207c8d963b4f268a248271179c519cdf312f3bdb7ce488c3ba6dcf964596a1d"}
Sep 17 23:30:28.227877 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.227798    2582 scope.go:117] "RemoveContainer" containerID="c8d7450b3226079b35c4c0173b259f51c27f908949ba907cb2392e9fab4608dc"
Sep 17 23:30:28.229554 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.229528    2582 generic.go:356] "Generic (PLEG): container finished" podID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerID="fb9525c8c49e4e068c9acc0023744917956a8c8a65de4d67c83ca1f89208546a" exitCode=0
Sep 17 23:30:28.229660 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.229619    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" event={"ID":"a65169dc-7302-4d5f-8fd1-a09b169f5229","Type":"ContainerDied","Data":"fb9525c8c49e4e068c9acc0023744917956a8c8a65de4d67c83ca1f89208546a"}
Sep 17 23:30:28.231417 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.231395    2582 generic.go:356] "Generic (PLEG): container finished" podID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerID="91cdc5f4e684804217d67e0a7343e18faa0f698478dad0fdf370120287c70376" exitCode=0
Sep 17 23:30:28.231529 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.231428    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" event={"ID":"5f4f16b7-e915-4ec5-8540-c4c38e713afe","Type":"ContainerDied","Data":"91cdc5f4e684804217d67e0a7343e18faa0f698478dad0fdf370120287c70376"}
Sep 17 23:30:28.237500 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.237391    2582 scope.go:117] "RemoveContainer" containerID="1c9498c1289f40dc526dab79da4c6bb4ec1727014512488207182b3a8be04ebb"
Sep 17 23:30:28.250897 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.250867    2582 scope.go:117] "RemoveContainer" containerID="368f0a4d6fc5239def1e039b2efdfa8d1fdb357416e87b671bf9443a6eb50247"
Sep 17 23:30:28.267109 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.267066    2582 scope.go:117] "RemoveContainer" containerID="c8d7450b3226079b35c4c0173b259f51c27f908949ba907cb2392e9fab4608dc"
Sep 17 23:30:28.267526 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:30:28.267496    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d7450b3226079b35c4c0173b259f51c27f908949ba907cb2392e9fab4608dc\": container with ID starting with c8d7450b3226079b35c4c0173b259f51c27f908949ba907cb2392e9fab4608dc not found: ID does not exist" containerID="c8d7450b3226079b35c4c0173b259f51c27f908949ba907cb2392e9fab4608dc"
Sep 17 23:30:28.267687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.267539    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d7450b3226079b35c4c0173b259f51c27f908949ba907cb2392e9fab4608dc"} err="failed to get container status \"c8d7450b3226079b35c4c0173b259f51c27f908949ba907cb2392e9fab4608dc\": rpc error: code = NotFound desc = could not find container \"c8d7450b3226079b35c4c0173b259f51c27f908949ba907cb2392e9fab4608dc\": container with ID starting with c8d7450b3226079b35c4c0173b259f51c27f908949ba907cb2392e9fab4608dc not found: ID does not exist"
Sep 17 23:30:28.267687 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.267576    2582 scope.go:117] "RemoveContainer" containerID="1c9498c1289f40dc526dab79da4c6bb4ec1727014512488207182b3a8be04ebb"
Sep 17 23:30:28.267945 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:30:28.267915    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9498c1289f40dc526dab79da4c6bb4ec1727014512488207182b3a8be04ebb\": container with ID starting with 1c9498c1289f40dc526dab79da4c6bb4ec1727014512488207182b3a8be04ebb not found: ID does not exist" containerID="1c9498c1289f40dc526dab79da4c6bb4ec1727014512488207182b3a8be04ebb"
Sep 17 23:30:28.268037 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.267952    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9498c1289f40dc526dab79da4c6bb4ec1727014512488207182b3a8be04ebb"} err="failed to get container status \"1c9498c1289f40dc526dab79da4c6bb4ec1727014512488207182b3a8be04ebb\": rpc error: code = NotFound desc = could not find container \"1c9498c1289f40dc526dab79da4c6bb4ec1727014512488207182b3a8be04ebb\": container with ID starting with 1c9498c1289f40dc526dab79da4c6bb4ec1727014512488207182b3a8be04ebb not found: ID does not exist"
Sep 17 23:30:28.268037 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.267970    2582 scope.go:117] "RemoveContainer" containerID="368f0a4d6fc5239def1e039b2efdfa8d1fdb357416e87b671bf9443a6eb50247"
Sep 17 23:30:28.268382 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:30:28.268354    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368f0a4d6fc5239def1e039b2efdfa8d1fdb357416e87b671bf9443a6eb50247\": container with ID starting with 368f0a4d6fc5239def1e039b2efdfa8d1fdb357416e87b671bf9443a6eb50247 not found: ID does not exist" containerID="368f0a4d6fc5239def1e039b2efdfa8d1fdb357416e87b671bf9443a6eb50247"
Sep 17 23:30:28.268498 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.268381    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368f0a4d6fc5239def1e039b2efdfa8d1fdb357416e87b671bf9443a6eb50247"} err="failed to get container status \"368f0a4d6fc5239def1e039b2efdfa8d1fdb357416e87b671bf9443a6eb50247\": rpc error: code = NotFound desc = could not find container \"368f0a4d6fc5239def1e039b2efdfa8d1fdb357416e87b671bf9443a6eb50247\": container with ID starting with 368f0a4d6fc5239def1e039b2efdfa8d1fdb357416e87b671bf9443a6eb50247 not found: ID does not exist"
Sep 17 23:30:28.286773 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.286745    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"]
Sep 17 23:30:28.293926 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:28.293739    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a284b-predictor-684c656bd9-r5g2l"]
Sep 17 23:30:29.238560 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:29.238525    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" event={"ID":"a65169dc-7302-4d5f-8fd1-a09b169f5229","Type":"ContainerStarted","Data":"d0a07658a88a4b4e8048a40481cc07a05a16b6c5699c1bbbf806ce56e6e6b050"}
Sep 17 23:30:29.239012 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:29.238606    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" event={"ID":"a65169dc-7302-4d5f-8fd1-a09b169f5229","Type":"ContainerStarted","Data":"663d29eafdfe0a5a8668853897cdced7caec4290a5a61f32b31fdd68cab2949e"}
Sep 17 23:30:29.239012 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:29.238867    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:29.240629 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:29.240594    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" event={"ID":"5f4f16b7-e915-4ec5-8540-c4c38e713afe","Type":"ContainerStarted","Data":"d938fe57328b142e178a773c90a0f45531368e56af196a118000fb41a8672135"}
Sep 17 23:30:29.240752 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:29.240652    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" event={"ID":"5f4f16b7-e915-4ec5-8540-c4c38e713afe","Type":"ContainerStarted","Data":"7f93e6afd885b48ffd2cc6f9e50f33080f9afdbdfe72d038272548d5c23e8f0b"}
Sep 17 23:30:29.240895 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:29.240877    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:29.262399 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:29.262320    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" podStartSLOduration=3.58179378 podStartE2EDuration="7.262304825s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.680511045s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:29.260862416 +0000 UTC m=+1183.947934987" watchObservedRunningTime="2026-09-17 23:30:29.262304825 +0000 UTC m=+1183.949377403"
Sep 17 23:30:29.282120 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:29.282065    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" podStartSLOduration=3.360407007 podStartE2EDuration="7.282049542s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.921642535s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:29.280709744 +0000 UTC m=+1183.967782343" watchObservedRunningTime="2026-09-17 23:30:29.282049542 +0000 UTC m=+1183.969122119"
Sep 17 23:30:29.393956 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:29.393922    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01e68429-e219-40fa-b661-087ca5136e5e" path="/var/lib/kubelet/pods/01e68429-e219-40fa-b661-087ca5136e5e/volumes"
Sep 17 23:30:30.245812 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:30.245771    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:30.246248 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:30.245863    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:30.246809 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:30.246785    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 23:30:30.246869 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:30.246785    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:30:31.180790 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.180763    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:30:31.251773 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.251676    2582 generic.go:356] "Generic (PLEG): container finished" podID="03882ad6-d499-41f5-aece-c3c5ea157867" containerID="46d67d0e31c1dcf856c79dee500b1ff806e3e7c2a893c5cb230ffa77e0b5a3f7" exitCode=0
Sep 17 23:30:31.251773 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.251732    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"
Sep 17 23:30:31.252338 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.251767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" event={"ID":"03882ad6-d499-41f5-aece-c3c5ea157867","Type":"ContainerDied","Data":"46d67d0e31c1dcf856c79dee500b1ff806e3e7c2a893c5cb230ffa77e0b5a3f7"}
Sep 17 23:30:31.252338 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.251870    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm" event={"ID":"03882ad6-d499-41f5-aece-c3c5ea157867","Type":"ContainerDied","Data":"0b4a03751bd585f1ea917fe8239f693a9d6fadb9a55128dfd7e489ebae4a0afd"}
Sep 17 23:30:31.252338 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.251914    2582 scope.go:117] "RemoveContainer" containerID="ede4de5eb70b7370e978d708b9948ac994b6a00bc48e15a0a43b34f1eda5c218"
Sep 17 23:30:31.252472 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.252391    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 23:30:31.252472 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.252393    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:30:31.260479 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.260459    2582 scope.go:117] "RemoveContainer" containerID="46d67d0e31c1dcf856c79dee500b1ff806e3e7c2a893c5cb230ffa77e0b5a3f7"
Sep 17 23:30:31.268471 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.268451    2582 scope.go:117] "RemoveContainer" containerID="e000888896a0bc102f22fda0d92c879e4f06cdd9837985d0a2794d06835328df"
Sep 17 23:30:31.277160 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.277140    2582 scope.go:117] "RemoveContainer" containerID="ede4de5eb70b7370e978d708b9948ac994b6a00bc48e15a0a43b34f1eda5c218"
Sep 17 23:30:31.277483 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:30:31.277458    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede4de5eb70b7370e978d708b9948ac994b6a00bc48e15a0a43b34f1eda5c218\": container with ID starting with ede4de5eb70b7370e978d708b9948ac994b6a00bc48e15a0a43b34f1eda5c218 not found: ID does not exist" containerID="ede4de5eb70b7370e978d708b9948ac994b6a00bc48e15a0a43b34f1eda5c218"
Sep 17 23:30:31.277582 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.277494    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede4de5eb70b7370e978d708b9948ac994b6a00bc48e15a0a43b34f1eda5c218"} err="failed to get container status \"ede4de5eb70b7370e978d708b9948ac994b6a00bc48e15a0a43b34f1eda5c218\": rpc error: code = NotFound desc = could not find container \"ede4de5eb70b7370e978d708b9948ac994b6a00bc48e15a0a43b34f1eda5c218\": container with ID starting with ede4de5eb70b7370e978d708b9948ac994b6a00bc48e15a0a43b34f1eda5c218 not found: ID does not exist"
Sep 17 23:30:31.277582 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.277514    2582 scope.go:117] "RemoveContainer" containerID="46d67d0e31c1dcf856c79dee500b1ff806e3e7c2a893c5cb230ffa77e0b5a3f7"
Sep 17 23:30:31.277830 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:30:31.277812    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d67d0e31c1dcf856c79dee500b1ff806e3e7c2a893c5cb230ffa77e0b5a3f7\": container with ID starting with 46d67d0e31c1dcf856c79dee500b1ff806e3e7c2a893c5cb230ffa77e0b5a3f7 not found: ID does not exist" containerID="46d67d0e31c1dcf856c79dee500b1ff806e3e7c2a893c5cb230ffa77e0b5a3f7"
Sep 17 23:30:31.277895 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.277860    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d67d0e31c1dcf856c79dee500b1ff806e3e7c2a893c5cb230ffa77e0b5a3f7"} err="failed to get container status \"46d67d0e31c1dcf856c79dee500b1ff806e3e7c2a893c5cb230ffa77e0b5a3f7\": rpc error: code = NotFound desc = could not find container \"46d67d0e31c1dcf856c79dee500b1ff806e3e7c2a893c5cb230ffa77e0b5a3f7\": container with ID starting with 46d67d0e31c1dcf856c79dee500b1ff806e3e7c2a893c5cb230ffa77e0b5a3f7 not found: ID does not exist"
Sep 17 23:30:31.277895 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.277878    2582 scope.go:117] "RemoveContainer" containerID="e000888896a0bc102f22fda0d92c879e4f06cdd9837985d0a2794d06835328df"
Sep 17 23:30:31.278160 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:30:31.278134    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e000888896a0bc102f22fda0d92c879e4f06cdd9837985d0a2794d06835328df\": container with ID starting with e000888896a0bc102f22fda0d92c879e4f06cdd9837985d0a2794d06835328df not found: ID does not exist" containerID="e000888896a0bc102f22fda0d92c879e4f06cdd9837985d0a2794d06835328df"
Sep 17 23:30:31.278233 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.278156    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e000888896a0bc102f22fda0d92c879e4f06cdd9837985d0a2794d06835328df"} err="failed to get container status \"e000888896a0bc102f22fda0d92c879e4f06cdd9837985d0a2794d06835328df\": rpc error: code = NotFound desc = could not find container \"e000888896a0bc102f22fda0d92c879e4f06cdd9837985d0a2794d06835328df\": container with ID starting with e000888896a0bc102f22fda0d92c879e4f06cdd9837985d0a2794d06835328df not found: ID does not exist"
Sep 17 23:30:31.281417 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.281400    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03882ad6-d499-41f5-aece-c3c5ea157867-kube-rbac-proxy-sar-config\") pod \"03882ad6-d499-41f5-aece-c3c5ea157867\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") "
Sep 17 23:30:31.281473 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.281452    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrv7x\" (UniqueName: \"kubernetes.io/projected/03882ad6-d499-41f5-aece-c3c5ea157867-kube-api-access-hrv7x\") pod \"03882ad6-d499-41f5-aece-c3c5ea157867\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") "
Sep 17 23:30:31.281508 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.281485    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03882ad6-d499-41f5-aece-c3c5ea157867-cabundle-cert\") pod \"03882ad6-d499-41f5-aece-c3c5ea157867\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") "
Sep 17 23:30:31.281542 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.281511    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03882ad6-d499-41f5-aece-c3c5ea157867-kserve-provision-location\") pod \"03882ad6-d499-41f5-aece-c3c5ea157867\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") "
Sep 17 23:30:31.281587 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.281540    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03882ad6-d499-41f5-aece-c3c5ea157867-proxy-tls\") pod \"03882ad6-d499-41f5-aece-c3c5ea157867\" (UID: \"03882ad6-d499-41f5-aece-c3c5ea157867\") "
Sep 17 23:30:31.282186 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.281791    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03882ad6-d499-41f5-aece-c3c5ea157867-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "03882ad6-d499-41f5-aece-c3c5ea157867" (UID: "03882ad6-d499-41f5-aece-c3c5ea157867"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:30:31.282186 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.281827    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03882ad6-d499-41f5-aece-c3c5ea157867-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "03882ad6-d499-41f5-aece-c3c5ea157867" (UID: "03882ad6-d499-41f5-aece-c3c5ea157867"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:31.282186 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.281947    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03882ad6-d499-41f5-aece-c3c5ea157867-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:30:31.282186 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.281965    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03882ad6-d499-41f5-aece-c3c5ea157867-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:30:31.282186 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.282175    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03882ad6-d499-41f5-aece-c3c5ea157867-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "03882ad6-d499-41f5-aece-c3c5ea157867" (UID: "03882ad6-d499-41f5-aece-c3c5ea157867"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:30:31.283895 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.283870    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03882ad6-d499-41f5-aece-c3c5ea157867-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03882ad6-d499-41f5-aece-c3c5ea157867" (UID: "03882ad6-d499-41f5-aece-c3c5ea157867"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:30:31.283895 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.283882    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03882ad6-d499-41f5-aece-c3c5ea157867-kube-api-access-hrv7x" (OuterVolumeSpecName: "kube-api-access-hrv7x") pod "03882ad6-d499-41f5-aece-c3c5ea157867" (UID: "03882ad6-d499-41f5-aece-c3c5ea157867"). InnerVolumeSpecName "kube-api-access-hrv7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:30:31.383127 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.383087    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrv7x\" (UniqueName: \"kubernetes.io/projected/03882ad6-d499-41f5-aece-c3c5ea157867-kube-api-access-hrv7x\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:30:31.383127 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.383121    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03882ad6-d499-41f5-aece-c3c5ea157867-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:30:31.383127 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.383130    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03882ad6-d499-41f5-aece-c3c5ea157867-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:30:31.569239 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.569138    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"]
Sep 17 23:30:31.575613 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:31.575581    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-60b0d-predictor-55c6df5959-c8crm"]
Sep 17 23:30:33.394795 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:33.394763    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" path="/var/lib/kubelet/pods/03882ad6-d499-41f5-aece-c3c5ea157867/volumes"
Sep 17 23:30:36.256934 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:36.256894    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:30:36.257419 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:36.257281    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:30:36.257419 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:36.257365    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 23:30:36.257821 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:36.257795    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:30:46.258335 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:46.258292    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:30:46.258760 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:46.258292    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 23:30:56.257365 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:56.257328    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 23:30:56.257770 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:30:56.257746    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:31:06.257402 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:06.257360    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 23:31:06.257829 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:06.257772    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:31:16.257730 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:16.257687    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:31:16.258146 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:16.257687    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 23:31:26.258237 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:26.258195    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:31:26.258718 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:26.258195    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 23:31:36.257453 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:36.257371    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 23:31:36.257869 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:36.257640    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:31:46.258551 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:46.258518    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:31:46.260987 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:46.258624    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:31:46.390555 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:46.390522    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"]
Sep 17 23:31:46.543707 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:46.543625    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kube-rbac-proxy" containerID="cri-o://d0a07658a88a4b4e8048a40481cc07a05a16b6c5699c1bbbf806ce56e6e6b050" gracePeriod=30
Sep 17 23:31:46.543908 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:46.543600    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kserve-container" containerID="cri-o://663d29eafdfe0a5a8668853897cdced7caec4290a5a61f32b31fdd68cab2949e" gracePeriod=30
Sep 17 23:31:47.548705 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:47.548672    2582 generic.go:356] "Generic (PLEG): container finished" podID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerID="d0a07658a88a4b4e8048a40481cc07a05a16b6c5699c1bbbf806ce56e6e6b050" exitCode=2
Sep 17 23:31:47.549072 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:47.548725    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" event={"ID":"a65169dc-7302-4d5f-8fd1-a09b169f5229","Type":"ContainerDied","Data":"d0a07658a88a4b4e8048a40481cc07a05a16b6c5699c1bbbf806ce56e6e6b050"}
Sep 17 23:31:50.785499 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.785475    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:31:50.896635 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.896603    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65169dc-7302-4d5f-8fd1-a09b169f5229-cabundle-cert\") pod \"a65169dc-7302-4d5f-8fd1-a09b169f5229\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") "
Sep 17 23:31:50.896807 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.896645    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgrd2\" (UniqueName: \"kubernetes.io/projected/a65169dc-7302-4d5f-8fd1-a09b169f5229-kube-api-access-xgrd2\") pod \"a65169dc-7302-4d5f-8fd1-a09b169f5229\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") "
Sep 17 23:31:50.896807 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.896724    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65169dc-7302-4d5f-8fd1-a09b169f5229-kube-rbac-proxy-sar-config\") pod \"a65169dc-7302-4d5f-8fd1-a09b169f5229\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") "
Sep 17 23:31:50.896807 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.896763    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65169dc-7302-4d5f-8fd1-a09b169f5229-proxy-tls\") pod \"a65169dc-7302-4d5f-8fd1-a09b169f5229\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") "
Sep 17 23:31:50.896807 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.896786    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65169dc-7302-4d5f-8fd1-a09b169f5229-kserve-provision-location\") pod \"a65169dc-7302-4d5f-8fd1-a09b169f5229\" (UID: \"a65169dc-7302-4d5f-8fd1-a09b169f5229\") "
Sep 17 23:31:50.897121 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.897084    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65169dc-7302-4d5f-8fd1-a09b169f5229-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a65169dc-7302-4d5f-8fd1-a09b169f5229" (UID: "a65169dc-7302-4d5f-8fd1-a09b169f5229"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:31:50.897226 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.897120    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65169dc-7302-4d5f-8fd1-a09b169f5229-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a65169dc-7302-4d5f-8fd1-a09b169f5229" (UID: "a65169dc-7302-4d5f-8fd1-a09b169f5229"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:31:50.897226 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.897156    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65169dc-7302-4d5f-8fd1-a09b169f5229-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a65169dc-7302-4d5f-8fd1-a09b169f5229" (UID: "a65169dc-7302-4d5f-8fd1-a09b169f5229"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:50.898919 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.898893    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65169dc-7302-4d5f-8fd1-a09b169f5229-kube-api-access-xgrd2" (OuterVolumeSpecName: "kube-api-access-xgrd2") pod "a65169dc-7302-4d5f-8fd1-a09b169f5229" (UID: "a65169dc-7302-4d5f-8fd1-a09b169f5229"). InnerVolumeSpecName "kube-api-access-xgrd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:31:50.898995 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.898931    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65169dc-7302-4d5f-8fd1-a09b169f5229-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a65169dc-7302-4d5f-8fd1-a09b169f5229" (UID: "a65169dc-7302-4d5f-8fd1-a09b169f5229"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:31:50.997605 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.997560    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a65169dc-7302-4d5f-8fd1-a09b169f5229-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:31:50.997605 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.997601    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a65169dc-7302-4d5f-8fd1-a09b169f5229-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:31:50.997605 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.997610    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a65169dc-7302-4d5f-8fd1-a09b169f5229-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:31:50.997605 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.997619    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgrd2\" (UniqueName: \"kubernetes.io/projected/a65169dc-7302-4d5f-8fd1-a09b169f5229-kube-api-access-xgrd2\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:31:50.997898 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:50.997627    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a65169dc-7302-4d5f-8fd1-a09b169f5229-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:31:51.566095 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:51.566059    2582 generic.go:356] "Generic (PLEG): container finished" podID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerID="663d29eafdfe0a5a8668853897cdced7caec4290a5a61f32b31fdd68cab2949e" exitCode=0
Sep 17 23:31:51.566239 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:51.566116    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" event={"ID":"a65169dc-7302-4d5f-8fd1-a09b169f5229","Type":"ContainerDied","Data":"663d29eafdfe0a5a8668853897cdced7caec4290a5a61f32b31fdd68cab2949e"}
Sep 17 23:31:51.566239 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:51.566127    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"
Sep 17 23:31:51.566239 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:51.566189    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44" event={"ID":"a65169dc-7302-4d5f-8fd1-a09b169f5229","Type":"ContainerDied","Data":"f51db9769753e1d525f8e9b0eca5e0fae29fa8361c3bd636ffed71613cb73dfd"}
Sep 17 23:31:51.566239 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:51.566219    2582 scope.go:117] "RemoveContainer" containerID="d0a07658a88a4b4e8048a40481cc07a05a16b6c5699c1bbbf806ce56e6e6b050"
Sep 17 23:31:51.574277 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:51.574258    2582 scope.go:117] "RemoveContainer" containerID="663d29eafdfe0a5a8668853897cdced7caec4290a5a61f32b31fdd68cab2949e"
Sep 17 23:31:51.581351 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:51.581323    2582 scope.go:117] "RemoveContainer" containerID="fb9525c8c49e4e068c9acc0023744917956a8c8a65de4d67c83ca1f89208546a"
Sep 17 23:31:51.584189 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:51.584169    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"]
Sep 17 23:31:51.588067 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:51.588047    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-5hl44"]
Sep 17 23:31:51.589443 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:51.589430    2582 scope.go:117] "RemoveContainer" containerID="d0a07658a88a4b4e8048a40481cc07a05a16b6c5699c1bbbf806ce56e6e6b050"
Sep 17 23:31:51.589698 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:31:51.589681    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0a07658a88a4b4e8048a40481cc07a05a16b6c5699c1bbbf806ce56e6e6b050\": container with ID starting with d0a07658a88a4b4e8048a40481cc07a05a16b6c5699c1bbbf806ce56e6e6b050 not found: ID does not exist" containerID="d0a07658a88a4b4e8048a40481cc07a05a16b6c5699c1bbbf806ce56e6e6b050"
Sep 17 23:31:51.589757 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:51.589705    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0a07658a88a4b4e8048a40481cc07a05a16b6c5699c1bbbf806ce56e6e6b050"} err="failed to get container status \"d0a07658a88a4b4e8048a40481cc07a05a16b6c5699c1bbbf806ce56e6e6b050\": rpc error: code = NotFound desc = could not find container \"d0a07658a88a4b4e8048a40481cc07a05a16b6c5699c1bbbf806ce56e6e6b050\": container with ID starting with d0a07658a88a4b4e8048a40481cc07a05a16b6c5699c1bbbf806ce56e6e6b050 not found: ID does not exist"
Sep 17 23:31:51.589757 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:51.589720    2582 scope.go:117] "RemoveContainer" containerID="663d29eafdfe0a5a8668853897cdced7caec4290a5a61f32b31fdd68cab2949e"
Sep 17 23:31:51.590003 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:31:51.589985    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663d29eafdfe0a5a8668853897cdced7caec4290a5a61f32b31fdd68cab2949e\": container with ID starting with 663d29eafdfe0a5a8668853897cdced7caec4290a5a61f32b31fdd68cab2949e not found: ID does not exist" containerID="663d29eafdfe0a5a8668853897cdced7caec4290a5a61f32b31fdd68cab2949e"
Sep 17 23:31:51.590101 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:51.590009    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663d29eafdfe0a5a8668853897cdced7caec4290a5a61f32b31fdd68cab2949e"} err="failed to get container status \"663d29eafdfe0a5a8668853897cdced7caec4290a5a61f32b31fdd68cab2949e\": rpc error: code = NotFound desc = could not find container \"663d29eafdfe0a5a8668853897cdced7caec4290a5a61f32b31fdd68cab2949e\": container with ID starting with 663d29eafdfe0a5a8668853897cdced7caec4290a5a61f32b31fdd68cab2949e not found: ID does not exist"
Sep 17 23:31:51.590101 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:51.590025    2582 scope.go:117] "RemoveContainer" containerID="fb9525c8c49e4e068c9acc0023744917956a8c8a65de4d67c83ca1f89208546a"
Sep 17 23:31:51.590274 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:31:51.590260    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9525c8c49e4e068c9acc0023744917956a8c8a65de4d67c83ca1f89208546a\": container with ID starting with fb9525c8c49e4e068c9acc0023744917956a8c8a65de4d67c83ca1f89208546a not found: ID does not exist" containerID="fb9525c8c49e4e068c9acc0023744917956a8c8a65de4d67c83ca1f89208546a"
Sep 17 23:31:51.590308 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:51.590278    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9525c8c49e4e068c9acc0023744917956a8c8a65de4d67c83ca1f89208546a"} err="failed to get container status \"fb9525c8c49e4e068c9acc0023744917956a8c8a65de4d67c83ca1f89208546a\": rpc error: code = NotFound desc = could not find container \"fb9525c8c49e4e068c9acc0023744917956a8c8a65de4d67c83ca1f89208546a\": container with ID starting with fb9525c8c49e4e068c9acc0023744917956a8c8a65de4d67c83ca1f89208546a not found: ID does not exist"
Sep 17 23:31:52.543206 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:52.543176    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"]
Sep 17 23:31:52.543656 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:52.543513    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kserve-container" containerID="cri-o://7f93e6afd885b48ffd2cc6f9e50f33080f9afdbdfe72d038272548d5c23e8f0b" gracePeriod=30
Sep 17 23:31:52.543656 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:52.543519    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kube-rbac-proxy" containerID="cri-o://d938fe57328b142e178a773c90a0f45531368e56af196a118000fb41a8672135" gracePeriod=30
Sep 17 23:31:53.393879 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:53.393825    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" path="/var/lib/kubelet/pods/a65169dc-7302-4d5f-8fd1-a09b169f5229/volumes"
Sep 17 23:31:53.577635 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:53.577602    2582 generic.go:356] "Generic (PLEG): container finished" podID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerID="d938fe57328b142e178a773c90a0f45531368e56af196a118000fb41a8672135" exitCode=2
Sep 17 23:31:53.578011 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:53.577670    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" event={"ID":"5f4f16b7-e915-4ec5-8540-c4c38e713afe","Type":"ContainerDied","Data":"d938fe57328b142e178a773c90a0f45531368e56af196a118000fb41a8672135"}
Sep 17 23:31:56.253157 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:56.253114    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 23:31:56.258332 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:56.258308    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 23:31:56.886925 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:56.886897    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:31:56.939959 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:56.939896    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f4f16b7-e915-4ec5-8540-c4c38e713afe-proxy-tls\") pod \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") "
Sep 17 23:31:56.939959 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:56.939937    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f4f16b7-e915-4ec5-8540-c4c38e713afe-cabundle-cert\") pod \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") "
Sep 17 23:31:56.940117 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:56.939961    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kube-rbac-proxy-sar-config\") pod \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") "
Sep 17 23:31:56.940117 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:56.939980    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h88l2\" (UniqueName: \"kubernetes.io/projected/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kube-api-access-h88l2\") pod \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") "
Sep 17 23:31:56.940117 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:56.940099    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kserve-provision-location\") pod \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\" (UID: \"5f4f16b7-e915-4ec5-8540-c4c38e713afe\") "
Sep 17 23:31:56.940277 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:56.940252    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4f16b7-e915-4ec5-8540-c4c38e713afe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5f4f16b7-e915-4ec5-8540-c4c38e713afe" (UID: "5f4f16b7-e915-4ec5-8540-c4c38e713afe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:31:56.940333 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:56.940292    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5f4f16b7-e915-4ec5-8540-c4c38e713afe" (UID: "5f4f16b7-e915-4ec5-8540-c4c38e713afe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:31:56.940428 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:56.940411    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f4f16b7-e915-4ec5-8540-c4c38e713afe-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:31:56.940484 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:56.940431    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:31:56.940484 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:56.940437    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5f4f16b7-e915-4ec5-8540-c4c38e713afe" (UID: "5f4f16b7-e915-4ec5-8540-c4c38e713afe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:31:56.942230 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:56.942213    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f4f16b7-e915-4ec5-8540-c4c38e713afe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f4f16b7-e915-4ec5-8540-c4c38e713afe" (UID: "5f4f16b7-e915-4ec5-8540-c4c38e713afe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:31:56.942296 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:56.942281    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kube-api-access-h88l2" (OuterVolumeSpecName: "kube-api-access-h88l2") pod "5f4f16b7-e915-4ec5-8540-c4c38e713afe" (UID: "5f4f16b7-e915-4ec5-8540-c4c38e713afe"). InnerVolumeSpecName "kube-api-access-h88l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:31:57.041605 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.041573    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:31:57.041605 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.041598    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f4f16b7-e915-4ec5-8540-c4c38e713afe-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:31:57.041605 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.041612    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h88l2\" (UniqueName: \"kubernetes.io/projected/5f4f16b7-e915-4ec5-8540-c4c38e713afe-kube-api-access-h88l2\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:31:57.593713 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.593679    2582 generic.go:356] "Generic (PLEG): container finished" podID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerID="7f93e6afd885b48ffd2cc6f9e50f33080f9afdbdfe72d038272548d5c23e8f0b" exitCode=0
Sep 17 23:31:57.594108 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.593733    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" event={"ID":"5f4f16b7-e915-4ec5-8540-c4c38e713afe","Type":"ContainerDied","Data":"7f93e6afd885b48ffd2cc6f9e50f33080f9afdbdfe72d038272548d5c23e8f0b"}
Sep 17 23:31:57.594108 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.593750    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"
Sep 17 23:31:57.594108 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.593812    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r" event={"ID":"5f4f16b7-e915-4ec5-8540-c4c38e713afe","Type":"ContainerDied","Data":"d9452d9a03c9c4e1d633ba82506ecb418db1ffa93b9ce593e0b3aaaae0ca72e8"}
Sep 17 23:31:57.594108 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.593873    2582 scope.go:117] "RemoveContainer" containerID="d938fe57328b142e178a773c90a0f45531368e56af196a118000fb41a8672135"
Sep 17 23:31:57.602027 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.602006    2582 scope.go:117] "RemoveContainer" containerID="7f93e6afd885b48ffd2cc6f9e50f33080f9afdbdfe72d038272548d5c23e8f0b"
Sep 17 23:31:57.610788 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.610767    2582 scope.go:117] "RemoveContainer" containerID="91cdc5f4e684804217d67e0a7343e18faa0f698478dad0fdf370120287c70376"
Sep 17 23:31:57.618894 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.618878    2582 scope.go:117] "RemoveContainer" containerID="d938fe57328b142e178a773c90a0f45531368e56af196a118000fb41a8672135"
Sep 17 23:31:57.619146 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:31:57.619125    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d938fe57328b142e178a773c90a0f45531368e56af196a118000fb41a8672135\": container with ID starting with d938fe57328b142e178a773c90a0f45531368e56af196a118000fb41a8672135 not found: ID does not exist" containerID="d938fe57328b142e178a773c90a0f45531368e56af196a118000fb41a8672135"
Sep 17 23:31:57.619203 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.619154    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d938fe57328b142e178a773c90a0f45531368e56af196a118000fb41a8672135"} err="failed to get container status \"d938fe57328b142e178a773c90a0f45531368e56af196a118000fb41a8672135\": rpc error: code = NotFound desc = could not find container \"d938fe57328b142e178a773c90a0f45531368e56af196a118000fb41a8672135\": container with ID starting with d938fe57328b142e178a773c90a0f45531368e56af196a118000fb41a8672135 not found: ID does not exist"
Sep 17 23:31:57.619203 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.619174    2582 scope.go:117] "RemoveContainer" containerID="7f93e6afd885b48ffd2cc6f9e50f33080f9afdbdfe72d038272548d5c23e8f0b"
Sep 17 23:31:57.619426 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:31:57.619407    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f93e6afd885b48ffd2cc6f9e50f33080f9afdbdfe72d038272548d5c23e8f0b\": container with ID starting with 7f93e6afd885b48ffd2cc6f9e50f33080f9afdbdfe72d038272548d5c23e8f0b not found: ID does not exist" containerID="7f93e6afd885b48ffd2cc6f9e50f33080f9afdbdfe72d038272548d5c23e8f0b"
Sep 17 23:31:57.619470 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.619434    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f93e6afd885b48ffd2cc6f9e50f33080f9afdbdfe72d038272548d5c23e8f0b"} err="failed to get container status \"7f93e6afd885b48ffd2cc6f9e50f33080f9afdbdfe72d038272548d5c23e8f0b\": rpc error: code = NotFound desc = could not find container \"7f93e6afd885b48ffd2cc6f9e50f33080f9afdbdfe72d038272548d5c23e8f0b\": container with ID starting with 7f93e6afd885b48ffd2cc6f9e50f33080f9afdbdfe72d038272548d5c23e8f0b not found: ID does not exist"
Sep 17 23:31:57.619470 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.619450    2582 scope.go:117] "RemoveContainer" containerID="91cdc5f4e684804217d67e0a7343e18faa0f698478dad0fdf370120287c70376"
Sep 17 23:31:57.619691 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:31:57.619674    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91cdc5f4e684804217d67e0a7343e18faa0f698478dad0fdf370120287c70376\": container with ID starting with 91cdc5f4e684804217d67e0a7343e18faa0f698478dad0fdf370120287c70376 not found: ID does not exist" containerID="91cdc5f4e684804217d67e0a7343e18faa0f698478dad0fdf370120287c70376"
Sep 17 23:31:57.619743 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.619695    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91cdc5f4e684804217d67e0a7343e18faa0f698478dad0fdf370120287c70376"} err="failed to get container status \"91cdc5f4e684804217d67e0a7343e18faa0f698478dad0fdf370120287c70376\": rpc error: code = NotFound desc = could not find container \"91cdc5f4e684804217d67e0a7343e18faa0f698478dad0fdf370120287c70376\": container with ID starting with 91cdc5f4e684804217d67e0a7343e18faa0f698478dad0fdf370120287c70376 not found: ID does not exist"
Sep 17 23:31:57.646580 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.646549    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"]
Sep 17 23:31:57.651019 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.650998    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7cfc7d4c75-hwz2r"]
Sep 17 23:31:57.917448 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.917412    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"]
Sep 17 23:31:57.917886 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.917865    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="storage-initializer"
Sep 17 23:31:57.917886 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.917885    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="storage-initializer"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.917897    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="storage-initializer"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.917907    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="storage-initializer"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.917918    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kube-rbac-proxy"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.917928    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kube-rbac-proxy"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.917945    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="storage-initializer"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.917953    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="storage-initializer"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.917964    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.917971    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.917988    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kserve-container"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.917996    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kserve-container"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918009    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="storage-initializer"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918017    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="storage-initializer"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918025    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kube-rbac-proxy"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918033    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kube-rbac-proxy"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918047    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kube-rbac-proxy"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918054    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kube-rbac-proxy"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918065    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kserve-container"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918074    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kserve-container"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918083    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kserve-container"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918092    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kserve-container"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918101    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kube-rbac-proxy"
Sep 17 23:31:57.918107 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918108    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kube-rbac-proxy"
Sep 17 23:31:57.918993 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918175    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kube-rbac-proxy"
Sep 17 23:31:57.918993 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918187    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kube-rbac-proxy"
Sep 17 23:31:57.918993 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918195    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kube-rbac-proxy"
Sep 17 23:31:57.918993 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918202    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" containerName="kserve-container"
Sep 17 23:31:57.918993 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918211    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kube-rbac-proxy"
Sep 17 23:31:57.918993 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918219    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a65169dc-7302-4d5f-8fd1-a09b169f5229" containerName="kserve-container"
Sep 17 23:31:57.918993 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918226    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e68429-e219-40fa-b661-087ca5136e5e" containerName="kserve-container"
Sep 17 23:31:57.918993 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.918235    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="03882ad6-d499-41f5-aece-c3c5ea157867" containerName="kserve-container"
Sep 17 23:31:57.922770 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.922754    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"]
Sep 17 23:31:57.922888 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.922871    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:57.926297 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.926278    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:57.930411 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.930193    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:31:57.930411 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.930235    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f2cbg\""
Sep 17 23:31:57.930544 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.930451    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:31:57.930627 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.930607    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 23:31:57.930731 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.930640    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 23:31:57.930812 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.930668    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:31:57.930909 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.930815    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:31:57.930909 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.930671    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 23:31:57.936079 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.936059    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"]
Sep 17 23:31:57.937587 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:57.937569    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"]
Sep 17 23:31:58.047884 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.047826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsqn5\" (UniqueName: \"kubernetes.io/projected/976f73c4-f31b-492d-a29a-43ab1e297ccf-kube-api-access-dsqn5\") pod \"isvc-canary-promote-predictor-fc78f5cb6-65h2m\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.048067 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.047893    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976f73c4-f31b-492d-a29a-43ab1e297ccf-proxy-tls\") pod \"isvc-canary-promote-predictor-fc78f5cb6-65h2m\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.048067 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.047919    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w28sd\" (UniqueName: \"kubernetes.io/projected/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kube-api-access-w28sd\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-975sd\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.048067 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.047938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/976f73c4-f31b-492d-a29a-43ab1e297ccf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-fc78f5cb6-65h2m\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.048067 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.047977    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/976f73c4-f31b-492d-a29a-43ab1e297ccf-kserve-provision-location\") pod \"isvc-canary-promote-predictor-fc78f5cb6-65h2m\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.048067 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.048028    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-975sd\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.048067 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.048053    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c072fa1-41f8-49e0-8bc9-596d32061cf7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-975sd\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.048332 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.048106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c072fa1-41f8-49e0-8bc9-596d32061cf7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-975sd\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.048332 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.048143    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/976f73c4-f31b-492d-a29a-43ab1e297ccf-cabundle-cert\") pod \"isvc-canary-promote-predictor-fc78f5cb6-65h2m\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.048332 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.048173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-975sd\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.148758 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.148729    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/976f73c4-f31b-492d-a29a-43ab1e297ccf-kserve-provision-location\") pod \"isvc-canary-promote-predictor-fc78f5cb6-65h2m\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.148945 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.148767    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-975sd\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.148945 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.148788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c072fa1-41f8-49e0-8bc9-596d32061cf7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-975sd\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.148945 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.148807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c072fa1-41f8-49e0-8bc9-596d32061cf7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-975sd\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.149117 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.149057    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/976f73c4-f31b-492d-a29a-43ab1e297ccf-cabundle-cert\") pod \"isvc-canary-promote-predictor-fc78f5cb6-65h2m\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.149117 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.149095    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-975sd\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.149219 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.149127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsqn5\" (UniqueName: \"kubernetes.io/projected/976f73c4-f31b-492d-a29a-43ab1e297ccf-kube-api-access-dsqn5\") pod \"isvc-canary-promote-predictor-fc78f5cb6-65h2m\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.149219 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.149161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976f73c4-f31b-492d-a29a-43ab1e297ccf-proxy-tls\") pod \"isvc-canary-promote-predictor-fc78f5cb6-65h2m\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.149219 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.149189    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/976f73c4-f31b-492d-a29a-43ab1e297ccf-kserve-provision-location\") pod \"isvc-canary-promote-predictor-fc78f5cb6-65h2m\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.149219 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.149208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w28sd\" (UniqueName: \"kubernetes.io/projected/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kube-api-access-w28sd\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-975sd\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.149419 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.149261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/976f73c4-f31b-492d-a29a-43ab1e297ccf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-fc78f5cb6-65h2m\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.149613 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.149587    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c072fa1-41f8-49e0-8bc9-596d32061cf7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-975sd\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.149681 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.149610    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-975sd\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.149727 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.149713    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-975sd\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.149781 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.149758    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/976f73c4-f31b-492d-a29a-43ab1e297ccf-cabundle-cert\") pod \"isvc-canary-promote-predictor-fc78f5cb6-65h2m\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.149965 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.149942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/976f73c4-f31b-492d-a29a-43ab1e297ccf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-fc78f5cb6-65h2m\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.152122 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.152099    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976f73c4-f31b-492d-a29a-43ab1e297ccf-proxy-tls\") pod \"isvc-canary-promote-predictor-fc78f5cb6-65h2m\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.152122 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.152109    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c072fa1-41f8-49e0-8bc9-596d32061cf7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-975sd\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.157425 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.157401    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w28sd\" (UniqueName: \"kubernetes.io/projected/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kube-api-access-w28sd\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-975sd\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.158092 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.158070    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsqn5\" (UniqueName: \"kubernetes.io/projected/976f73c4-f31b-492d-a29a-43ab1e297ccf-kube-api-access-dsqn5\") pod \"isvc-canary-promote-predictor-fc78f5cb6-65h2m\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.235870 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.235784    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:31:58.242954 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.242933    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:31:58.511978 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.511949    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"]
Sep 17 23:31:58.514412 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:31:58.514383    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c072fa1_41f8_49e0_8bc9_596d32061cf7.slice/crio-3f078c4ce71d4d6b8bd559027c8cfe1edc17f929822c0486dac7d52aa59e79ec WatchSource:0}: Error finding container 3f078c4ce71d4d6b8bd559027c8cfe1edc17f929822c0486dac7d52aa59e79ec: Status 404 returned error can't find the container with id 3f078c4ce71d4d6b8bd559027c8cfe1edc17f929822c0486dac7d52aa59e79ec
Sep 17 23:31:58.515891 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.515866    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"]
Sep 17 23:31:58.517618 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:31:58.517599    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976f73c4_f31b_492d_a29a_43ab1e297ccf.slice/crio-623f3162680e9208824159635b4eb704d43409f07206bb7e1aac7e3caa981308 WatchSource:0}: Error finding container 623f3162680e9208824159635b4eb704d43409f07206bb7e1aac7e3caa981308: Status 404 returned error can't find the container with id 623f3162680e9208824159635b4eb704d43409f07206bb7e1aac7e3caa981308
Sep 17 23:31:58.600955 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.600917    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" event={"ID":"2c072fa1-41f8-49e0-8bc9-596d32061cf7","Type":"ContainerStarted","Data":"57f1f9ca43926ce3ff0ce73cdbff07cf79957c171b13cb75fda1bae6df26bfc0"}
Sep 17 23:31:58.601381 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.601002    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" event={"ID":"2c072fa1-41f8-49e0-8bc9-596d32061cf7","Type":"ContainerStarted","Data":"3f078c4ce71d4d6b8bd559027c8cfe1edc17f929822c0486dac7d52aa59e79ec"}
Sep 17 23:31:58.602561 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.602538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" event={"ID":"976f73c4-f31b-492d-a29a-43ab1e297ccf","Type":"ContainerStarted","Data":"71da6611212e954c67d4ef08a4b117a26ac77a7fc6d855a5b9d23b37fb293d9c"}
Sep 17 23:31:58.602688 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:58.602591    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" event={"ID":"976f73c4-f31b-492d-a29a-43ab1e297ccf","Type":"ContainerStarted","Data":"623f3162680e9208824159635b4eb704d43409f07206bb7e1aac7e3caa981308"}
Sep 17 23:31:59.393960 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:31:59.393926    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f4f16b7-e915-4ec5-8540-c4c38e713afe" path="/var/lib/kubelet/pods/5f4f16b7-e915-4ec5-8540-c4c38e713afe/volumes"
Sep 17 23:32:02.618548 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:02.618519    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerID="57f1f9ca43926ce3ff0ce73cdbff07cf79957c171b13cb75fda1bae6df26bfc0" exitCode=0
Sep 17 23:32:02.618964 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:02.618590    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" event={"ID":"2c072fa1-41f8-49e0-8bc9-596d32061cf7","Type":"ContainerDied","Data":"57f1f9ca43926ce3ff0ce73cdbff07cf79957c171b13cb75fda1bae6df26bfc0"}
Sep 17 23:32:02.620288 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:02.620268    2582 generic.go:356] "Generic (PLEG): container finished" podID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerID="71da6611212e954c67d4ef08a4b117a26ac77a7fc6d855a5b9d23b37fb293d9c" exitCode=0
Sep 17 23:32:02.620388 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:02.620341    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" event={"ID":"976f73c4-f31b-492d-a29a-43ab1e297ccf","Type":"ContainerDied","Data":"71da6611212e954c67d4ef08a4b117a26ac77a7fc6d855a5b9d23b37fb293d9c"}
Sep 17 23:32:03.627682 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:03.627644    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" event={"ID":"2c072fa1-41f8-49e0-8bc9-596d32061cf7","Type":"ContainerStarted","Data":"743e48e7d864b4ec5f291549a405b1001144edbf08cdc703c2fbe1b2faa04aae"}
Sep 17 23:32:03.628114 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:03.627713    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" event={"ID":"2c072fa1-41f8-49e0-8bc9-596d32061cf7","Type":"ContainerStarted","Data":"8eb11baf7dd2f5600eddc5c61c9099a34bba33a36cb00965b1bc4d695cd16d13"}
Sep 17 23:32:03.628114 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:03.627975    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:32:03.629677 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:03.629655    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" event={"ID":"976f73c4-f31b-492d-a29a-43ab1e297ccf","Type":"ContainerStarted","Data":"3106e41f7e787598ddc9f7209ee777c8916a979d84a21278fcfcff6a59bbcf2d"}
Sep 17 23:32:03.629780 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:03.629701    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" event={"ID":"976f73c4-f31b-492d-a29a-43ab1e297ccf","Type":"ContainerStarted","Data":"6de25a70151605b4b4c88f7aa7ec0913f200724d161ef381443083ed93b10f64"}
Sep 17 23:32:03.629898 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:03.629883    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:32:03.653073 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:03.653027    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" podStartSLOduration=2.776934364 podStartE2EDuration="6.653011472s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.876077108s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:03.652111251 +0000 UTC m=+1278.339183851" watchObservedRunningTime="2026-09-17 23:32:03.653011472 +0000 UTC m=+1278.340084047"
Sep 17 23:32:03.675489 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:03.675433    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" podStartSLOduration=2.662565646 podStartE2EDuration="6.675411212s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.012845566s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:03.674493508 +0000 UTC m=+1278.361566084" watchObservedRunningTime="2026-09-17 23:32:03.675411212 +0000 UTC m=+1278.362483788"
Sep 17 23:32:04.634312 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:04.634276    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:32:04.634757 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:04.634344    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:32:04.635154 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:04.635127    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:32:04.635495 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:04.635473    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 23:32:05.638039 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:05.637997    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:32:05.638428 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:05.638104    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 23:32:10.643196 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:10.643166    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:32:10.643655 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:10.643260    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:32:10.643655 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:10.643607    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 23:32:10.643655 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:10.643634    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:32:20.644005 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:20.643966    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:32:20.644455 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:20.643966    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 23:32:30.643783 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:30.643743    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:32:30.644179 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:30.643743    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 23:32:40.644679 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:40.644633    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 23:32:40.645119 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:40.644633    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:32:50.643714 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:50.643673    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:32:50.644106 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:32:50.643674    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 23:33:00.644510 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:00.644425    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:33:00.644989 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:00.644421    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 23:33:10.644366 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:10.644319    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 23:33:10.644771 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:10.644324    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:33:20.645035 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:20.644996    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:33:20.645464 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:20.645419    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:33:20.784711 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:20.784678    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"]
Sep 17 23:33:20.929113 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:20.929024    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kserve-container" containerID="cri-o://6de25a70151605b4b4c88f7aa7ec0913f200724d161ef381443083ed93b10f64" gracePeriod=30
Sep 17 23:33:20.929113 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:20.929063    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kube-rbac-proxy" containerID="cri-o://3106e41f7e787598ddc9f7209ee777c8916a979d84a21278fcfcff6a59bbcf2d" gracePeriod=30
Sep 17 23:33:21.934554 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:21.934521    2582 generic.go:356] "Generic (PLEG): container finished" podID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerID="3106e41f7e787598ddc9f7209ee777c8916a979d84a21278fcfcff6a59bbcf2d" exitCode=2
Sep 17 23:33:21.934942 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:21.934585    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" event={"ID":"976f73c4-f31b-492d-a29a-43ab1e297ccf","Type":"ContainerDied","Data":"3106e41f7e787598ddc9f7209ee777c8916a979d84a21278fcfcff6a59bbcf2d"}
Sep 17 23:33:25.574425 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.574399    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:33:25.725509 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.725473    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsqn5\" (UniqueName: \"kubernetes.io/projected/976f73c4-f31b-492d-a29a-43ab1e297ccf-kube-api-access-dsqn5\") pod \"976f73c4-f31b-492d-a29a-43ab1e297ccf\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") "
Sep 17 23:33:25.725509 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.725510    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976f73c4-f31b-492d-a29a-43ab1e297ccf-proxy-tls\") pod \"976f73c4-f31b-492d-a29a-43ab1e297ccf\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") "
Sep 17 23:33:25.725780 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.725546    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/976f73c4-f31b-492d-a29a-43ab1e297ccf-kserve-provision-location\") pod \"976f73c4-f31b-492d-a29a-43ab1e297ccf\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") "
Sep 17 23:33:25.725780 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.725569    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/976f73c4-f31b-492d-a29a-43ab1e297ccf-cabundle-cert\") pod \"976f73c4-f31b-492d-a29a-43ab1e297ccf\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") "
Sep 17 23:33:25.725780 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.725622    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/976f73c4-f31b-492d-a29a-43ab1e297ccf-kube-rbac-proxy-sar-config\") pod \"976f73c4-f31b-492d-a29a-43ab1e297ccf\" (UID: \"976f73c4-f31b-492d-a29a-43ab1e297ccf\") "
Sep 17 23:33:25.726006 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.725918    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976f73c4-f31b-492d-a29a-43ab1e297ccf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "976f73c4-f31b-492d-a29a-43ab1e297ccf" (UID: "976f73c4-f31b-492d-a29a-43ab1e297ccf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:25.726071 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.726012    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976f73c4-f31b-492d-a29a-43ab1e297ccf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "976f73c4-f31b-492d-a29a-43ab1e297ccf" (UID: "976f73c4-f31b-492d-a29a-43ab1e297ccf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:25.726114 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.726079    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976f73c4-f31b-492d-a29a-43ab1e297ccf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "976f73c4-f31b-492d-a29a-43ab1e297ccf" (UID: "976f73c4-f31b-492d-a29a-43ab1e297ccf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:25.727792 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.727771    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976f73c4-f31b-492d-a29a-43ab1e297ccf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "976f73c4-f31b-492d-a29a-43ab1e297ccf" (UID: "976f73c4-f31b-492d-a29a-43ab1e297ccf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:25.727924 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.727906    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976f73c4-f31b-492d-a29a-43ab1e297ccf-kube-api-access-dsqn5" (OuterVolumeSpecName: "kube-api-access-dsqn5") pod "976f73c4-f31b-492d-a29a-43ab1e297ccf" (UID: "976f73c4-f31b-492d-a29a-43ab1e297ccf"). InnerVolumeSpecName "kube-api-access-dsqn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:25.826191 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.826141    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsqn5\" (UniqueName: \"kubernetes.io/projected/976f73c4-f31b-492d-a29a-43ab1e297ccf-kube-api-access-dsqn5\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:33:25.826191 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.826184    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/976f73c4-f31b-492d-a29a-43ab1e297ccf-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:33:25.826191 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.826196    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/976f73c4-f31b-492d-a29a-43ab1e297ccf-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:33:25.826191 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.826206    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/976f73c4-f31b-492d-a29a-43ab1e297ccf-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:33:25.826191 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.826214    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/976f73c4-f31b-492d-a29a-43ab1e297ccf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:33:25.954045 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.954005    2582 generic.go:356] "Generic (PLEG): container finished" podID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerID="6de25a70151605b4b4c88f7aa7ec0913f200724d161ef381443083ed93b10f64" exitCode=0
Sep 17 23:33:25.954215 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.954067    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" event={"ID":"976f73c4-f31b-492d-a29a-43ab1e297ccf","Type":"ContainerDied","Data":"6de25a70151605b4b4c88f7aa7ec0913f200724d161ef381443083ed93b10f64"}
Sep 17 23:33:25.954215 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.954076    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"
Sep 17 23:33:25.954215 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.954117    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m" event={"ID":"976f73c4-f31b-492d-a29a-43ab1e297ccf","Type":"ContainerDied","Data":"623f3162680e9208824159635b4eb704d43409f07206bb7e1aac7e3caa981308"}
Sep 17 23:33:25.954215 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.954142    2582 scope.go:117] "RemoveContainer" containerID="3106e41f7e787598ddc9f7209ee777c8916a979d84a21278fcfcff6a59bbcf2d"
Sep 17 23:33:25.962905 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.962883    2582 scope.go:117] "RemoveContainer" containerID="6de25a70151605b4b4c88f7aa7ec0913f200724d161ef381443083ed93b10f64"
Sep 17 23:33:25.970708 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.970688    2582 scope.go:117] "RemoveContainer" containerID="71da6611212e954c67d4ef08a4b117a26ac77a7fc6d855a5b9d23b37fb293d9c"
Sep 17 23:33:25.977134 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.977080    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"]
Sep 17 23:33:25.979467 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.979448    2582 scope.go:117] "RemoveContainer" containerID="3106e41f7e787598ddc9f7209ee777c8916a979d84a21278fcfcff6a59bbcf2d"
Sep 17 23:33:25.979786 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:33:25.979764    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3106e41f7e787598ddc9f7209ee777c8916a979d84a21278fcfcff6a59bbcf2d\": container with ID starting with 3106e41f7e787598ddc9f7209ee777c8916a979d84a21278fcfcff6a59bbcf2d not found: ID does not exist" containerID="3106e41f7e787598ddc9f7209ee777c8916a979d84a21278fcfcff6a59bbcf2d"
Sep 17 23:33:25.979889 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.979796    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3106e41f7e787598ddc9f7209ee777c8916a979d84a21278fcfcff6a59bbcf2d"} err="failed to get container status \"3106e41f7e787598ddc9f7209ee777c8916a979d84a21278fcfcff6a59bbcf2d\": rpc error: code = NotFound desc = could not find container \"3106e41f7e787598ddc9f7209ee777c8916a979d84a21278fcfcff6a59bbcf2d\": container with ID starting with 3106e41f7e787598ddc9f7209ee777c8916a979d84a21278fcfcff6a59bbcf2d not found: ID does not exist"
Sep 17 23:33:25.979889 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.979813    2582 scope.go:117] "RemoveContainer" containerID="6de25a70151605b4b4c88f7aa7ec0913f200724d161ef381443083ed93b10f64"
Sep 17 23:33:25.979889 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.979874    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-65h2m"]
Sep 17 23:33:25.980137 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:33:25.980118    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de25a70151605b4b4c88f7aa7ec0913f200724d161ef381443083ed93b10f64\": container with ID starting with 6de25a70151605b4b4c88f7aa7ec0913f200724d161ef381443083ed93b10f64 not found: ID does not exist" containerID="6de25a70151605b4b4c88f7aa7ec0913f200724d161ef381443083ed93b10f64"
Sep 17 23:33:25.980188 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.980142    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de25a70151605b4b4c88f7aa7ec0913f200724d161ef381443083ed93b10f64"} err="failed to get container status \"6de25a70151605b4b4c88f7aa7ec0913f200724d161ef381443083ed93b10f64\": rpc error: code = NotFound desc = could not find container \"6de25a70151605b4b4c88f7aa7ec0913f200724d161ef381443083ed93b10f64\": container with ID starting with 6de25a70151605b4b4c88f7aa7ec0913f200724d161ef381443083ed93b10f64 not found: ID does not exist"
Sep 17 23:33:25.980188 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.980158    2582 scope.go:117] "RemoveContainer" containerID="71da6611212e954c67d4ef08a4b117a26ac77a7fc6d855a5b9d23b37fb293d9c"
Sep 17 23:33:25.980411 ip-10-0-140-43 kubenswrapper[2582]: E0917 23:33:25.980393    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71da6611212e954c67d4ef08a4b117a26ac77a7fc6d855a5b9d23b37fb293d9c\": container with ID starting with 71da6611212e954c67d4ef08a4b117a26ac77a7fc6d855a5b9d23b37fb293d9c not found: ID does not exist" containerID="71da6611212e954c67d4ef08a4b117a26ac77a7fc6d855a5b9d23b37fb293d9c"
Sep 17 23:33:25.980453 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:25.980415    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71da6611212e954c67d4ef08a4b117a26ac77a7fc6d855a5b9d23b37fb293d9c"} err="failed to get container status \"71da6611212e954c67d4ef08a4b117a26ac77a7fc6d855a5b9d23b37fb293d9c\": rpc error: code = NotFound desc = could not find container \"71da6611212e954c67d4ef08a4b117a26ac77a7fc6d855a5b9d23b37fb293d9c\": container with ID starting with 71da6611212e954c67d4ef08a4b117a26ac77a7fc6d855a5b9d23b37fb293d9c not found: ID does not exist"
Sep 17 23:33:27.393521 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:33:27.393489    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" path="/var/lib/kubelet/pods/976f73c4-f31b-492d-a29a-43ab1e297ccf/volumes"
Sep 17 23:34:58.087283 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:34:58.087251    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"]
Sep 17 23:34:58.087792 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:34:58.087524    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kserve-container" containerID="cri-o://8eb11baf7dd2f5600eddc5c61c9099a34bba33a36cb00965b1bc4d695cd16d13" gracePeriod=30
Sep 17 23:34:58.087792 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:34:58.087607    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kube-rbac-proxy" containerID="cri-o://743e48e7d864b4ec5f291549a405b1001144edbf08cdc703c2fbe1b2faa04aae" gracePeriod=30
Sep 17 23:34:58.303990 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:34:58.303952    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerID="743e48e7d864b4ec5f291549a405b1001144edbf08cdc703c2fbe1b2faa04aae" exitCode=2
Sep 17 23:34:58.304131 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:34:58.304004    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" event={"ID":"2c072fa1-41f8-49e0-8bc9-596d32061cf7","Type":"ContainerDied","Data":"743e48e7d864b4ec5f291549a405b1001144edbf08cdc703c2fbe1b2faa04aae"}
Sep 17 23:35:00.638444 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:00.638407    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 23:35:00.643679 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:00.643655    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 23:35:02.322398 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.322368    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerID="8eb11baf7dd2f5600eddc5c61c9099a34bba33a36cb00965b1bc4d695cd16d13" exitCode=0
Sep 17 23:35:02.322734 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.322405    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" event={"ID":"2c072fa1-41f8-49e0-8bc9-596d32061cf7","Type":"ContainerDied","Data":"8eb11baf7dd2f5600eddc5c61c9099a34bba33a36cb00965b1bc4d695cd16d13"}
Sep 17 23:35:02.442869 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.442824    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:35:02.627065 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.626978    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c072fa1-41f8-49e0-8bc9-596d32061cf7-proxy-tls\") pod \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") "
Sep 17 23:35:02.627065 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.627027    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kserve-provision-location\") pod \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") "
Sep 17 23:35:02.627298 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.627076    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c072fa1-41f8-49e0-8bc9-596d32061cf7-cabundle-cert\") pod \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") "
Sep 17 23:35:02.627298 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.627110    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kube-rbac-proxy-sar-config\") pod \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") "
Sep 17 23:35:02.627298 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.627141    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w28sd\" (UniqueName: \"kubernetes.io/projected/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kube-api-access-w28sd\") pod \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\" (UID: \"2c072fa1-41f8-49e0-8bc9-596d32061cf7\") "
Sep 17 23:35:02.627457 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.627422    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c072fa1-41f8-49e0-8bc9-596d32061cf7" (UID: "2c072fa1-41f8-49e0-8bc9-596d32061cf7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:02.627500 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.627442    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c072fa1-41f8-49e0-8bc9-596d32061cf7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c072fa1-41f8-49e0-8bc9-596d32061cf7" (UID: "2c072fa1-41f8-49e0-8bc9-596d32061cf7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:35:02.627500 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.627455    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c072fa1-41f8-49e0-8bc9-596d32061cf7" (UID: "2c072fa1-41f8-49e0-8bc9-596d32061cf7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:35:02.629341 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.629318    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c072fa1-41f8-49e0-8bc9-596d32061cf7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c072fa1-41f8-49e0-8bc9-596d32061cf7" (UID: "2c072fa1-41f8-49e0-8bc9-596d32061cf7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:35:02.629409 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.629355    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kube-api-access-w28sd" (OuterVolumeSpecName: "kube-api-access-w28sd") pod "2c072fa1-41f8-49e0-8bc9-596d32061cf7" (UID: "2c072fa1-41f8-49e0-8bc9-596d32061cf7"). InnerVolumeSpecName "kube-api-access-w28sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:35:02.728335 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.728294    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c072fa1-41f8-49e0-8bc9-596d32061cf7-cabundle-cert\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:35:02.728335 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.728329    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:35:02.728335 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.728342    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w28sd\" (UniqueName: \"kubernetes.io/projected/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kube-api-access-w28sd\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:35:02.728569 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.728355    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c072fa1-41f8-49e0-8bc9-596d32061cf7-proxy-tls\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:35:02.728569 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:02.728369    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c072fa1-41f8-49e0-8bc9-596d32061cf7-kserve-provision-location\") on node \"ip-10-0-140-43.ec2.internal\" DevicePath \"\""
Sep 17 23:35:03.328399 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:03.328374    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"
Sep 17 23:35:03.328867 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:03.328369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd" event={"ID":"2c072fa1-41f8-49e0-8bc9-596d32061cf7","Type":"ContainerDied","Data":"3f078c4ce71d4d6b8bd559027c8cfe1edc17f929822c0486dac7d52aa59e79ec"}
Sep 17 23:35:03.328867 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:03.328534    2582 scope.go:117] "RemoveContainer" containerID="743e48e7d864b4ec5f291549a405b1001144edbf08cdc703c2fbe1b2faa04aae"
Sep 17 23:35:03.337610 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:03.337594    2582 scope.go:117] "RemoveContainer" containerID="8eb11baf7dd2f5600eddc5c61c9099a34bba33a36cb00965b1bc4d695cd16d13"
Sep 17 23:35:03.347551 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:03.347532    2582 scope.go:117] "RemoveContainer" containerID="57f1f9ca43926ce3ff0ce73cdbff07cf79957c171b13cb75fda1bae6df26bfc0"
Sep 17 23:35:03.352357 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:03.352335    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"]
Sep 17 23:35:03.360608 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:03.360585    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-975sd"]
Sep 17 23:35:03.397730 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:35:03.397700    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" path="/var/lib/kubelet/pods/2c072fa1-41f8-49e0-8bc9-596d32061cf7/volumes"
Sep 17 23:37:43.907905 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:43.907857    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-s7lg2_d48fd517-45d0-445c-836e-7045570e3385/global-pull-secret-syncer/0.log"
Sep 17 23:37:43.912261 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:43.912247    2582 ???:1] "http: TLS handshake error from 10.0.142.4:56084: EOF"
Sep 17 23:37:44.100400 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:44.100366    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mzjpw_0b41f4c0-9428-4c24-b1bc-725169912218/konnectivity-agent/0.log"
Sep 17 23:37:44.193048 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:44.192972    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-43.ec2.internal_ad31e4fc899beb2f71ca79970426a20a/haproxy/0.log"
Sep 17 23:37:48.567769 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:48.567739    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-qdg2c_2329ed7e-d5f9-46dc-a77c-14ef9dae6fc0/cluster-monitoring-operator/0.log"
Sep 17 23:37:48.981601 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:48.981568    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kklrc_6515a854-52fc-4271-ba77-f00f7e0dd25b/node-exporter/0.log"
Sep 17 23:37:49.009904 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:49.009879    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kklrc_6515a854-52fc-4271-ba77-f00f7e0dd25b/kube-rbac-proxy/0.log"
Sep 17 23:37:49.034696 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:49.034678    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kklrc_6515a854-52fc-4271-ba77-f00f7e0dd25b/init-textfile/0.log"
Sep 17 23:37:51.318915 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.318878    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-7jbxb_4bdccb4e-063e-4cda-8ed8-b36b36bf2643/networking-console-plugin/0.log"
Sep 17 23:37:51.967538 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.967499    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"]
Sep 17 23:37:51.967869 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.967830    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="storage-initializer"
Sep 17 23:37:51.967969 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.967872    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="storage-initializer"
Sep 17 23:37:51.967969 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.967887    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kserve-container"
Sep 17 23:37:51.967969 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.967894    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kserve-container"
Sep 17 23:37:51.967969 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.967904    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kube-rbac-proxy"
Sep 17 23:37:51.967969 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.967912    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kube-rbac-proxy"
Sep 17 23:37:51.967969 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.967932    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="storage-initializer"
Sep 17 23:37:51.967969 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.967941    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="storage-initializer"
Sep 17 23:37:51.967969 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.967955    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kube-rbac-proxy"
Sep 17 23:37:51.967969 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.967964    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kube-rbac-proxy"
Sep 17 23:37:51.968398 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.967975    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kserve-container"
Sep 17 23:37:51.968398 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.967984    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kserve-container"
Sep 17 23:37:51.968398 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.968066    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kserve-container"
Sep 17 23:37:51.968398 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.968079    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kserve-container"
Sep 17 23:37:51.968398 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.968089    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="976f73c4-f31b-492d-a29a-43ab1e297ccf" containerName="kube-rbac-proxy"
Sep 17 23:37:51.968398 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.968100    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c072fa1-41f8-49e0-8bc9-596d32061cf7" containerName="kube-rbac-proxy"
Sep 17 23:37:51.971099 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.971076    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:51.973644 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.973626    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wk5sv\"/\"kube-root-ca.crt\""
Sep 17 23:37:51.974780 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.974763    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wk5sv\"/\"default-dockercfg-g7v4k\""
Sep 17 23:37:51.974886 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.974765    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wk5sv\"/\"openshift-service-ca.crt\""
Sep 17 23:37:51.981627 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:51.981607    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"]
Sep 17 23:37:52.123036 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.123013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a089c387-9c45-4841-9a1e-7bbf0328ab29-proc\") pod \"perf-node-gather-daemonset-4nbqg\" (UID: \"a089c387-9c45-4841-9a1e-7bbf0328ab29\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.123153 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.123060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a089c387-9c45-4841-9a1e-7bbf0328ab29-lib-modules\") pod \"perf-node-gather-daemonset-4nbqg\" (UID: \"a089c387-9c45-4841-9a1e-7bbf0328ab29\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.123153 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.123085    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjgw2\" (UniqueName: \"kubernetes.io/projected/a089c387-9c45-4841-9a1e-7bbf0328ab29-kube-api-access-wjgw2\") pod \"perf-node-gather-daemonset-4nbqg\" (UID: \"a089c387-9c45-4841-9a1e-7bbf0328ab29\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.123153 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.123140    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a089c387-9c45-4841-9a1e-7bbf0328ab29-sys\") pod \"perf-node-gather-daemonset-4nbqg\" (UID: \"a089c387-9c45-4841-9a1e-7bbf0328ab29\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.123287 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.123184    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a089c387-9c45-4841-9a1e-7bbf0328ab29-podres\") pod \"perf-node-gather-daemonset-4nbqg\" (UID: \"a089c387-9c45-4841-9a1e-7bbf0328ab29\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.224094 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.224027    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a089c387-9c45-4841-9a1e-7bbf0328ab29-proc\") pod \"perf-node-gather-daemonset-4nbqg\" (UID: \"a089c387-9c45-4841-9a1e-7bbf0328ab29\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.224094 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.224080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a089c387-9c45-4841-9a1e-7bbf0328ab29-lib-modules\") pod \"perf-node-gather-daemonset-4nbqg\" (UID: \"a089c387-9c45-4841-9a1e-7bbf0328ab29\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.224265 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.224105    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjgw2\" (UniqueName: \"kubernetes.io/projected/a089c387-9c45-4841-9a1e-7bbf0328ab29-kube-api-access-wjgw2\") pod \"perf-node-gather-daemonset-4nbqg\" (UID: \"a089c387-9c45-4841-9a1e-7bbf0328ab29\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.224265 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.224135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a089c387-9c45-4841-9a1e-7bbf0328ab29-sys\") pod \"perf-node-gather-daemonset-4nbqg\" (UID: \"a089c387-9c45-4841-9a1e-7bbf0328ab29\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.224265 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.224145    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a089c387-9c45-4841-9a1e-7bbf0328ab29-proc\") pod \"perf-node-gather-daemonset-4nbqg\" (UID: \"a089c387-9c45-4841-9a1e-7bbf0328ab29\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.224265 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.224171    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a089c387-9c45-4841-9a1e-7bbf0328ab29-podres\") pod \"perf-node-gather-daemonset-4nbqg\" (UID: \"a089c387-9c45-4841-9a1e-7bbf0328ab29\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.224265 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.224209    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a089c387-9c45-4841-9a1e-7bbf0328ab29-sys\") pod \"perf-node-gather-daemonset-4nbqg\" (UID: \"a089c387-9c45-4841-9a1e-7bbf0328ab29\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.224265 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.224265    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a089c387-9c45-4841-9a1e-7bbf0328ab29-lib-modules\") pod \"perf-node-gather-daemonset-4nbqg\" (UID: \"a089c387-9c45-4841-9a1e-7bbf0328ab29\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.224484 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.224307    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a089c387-9c45-4841-9a1e-7bbf0328ab29-podres\") pod \"perf-node-gather-daemonset-4nbqg\" (UID: \"a089c387-9c45-4841-9a1e-7bbf0328ab29\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.233012 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.232985    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjgw2\" (UniqueName: \"kubernetes.io/projected/a089c387-9c45-4841-9a1e-7bbf0328ab29-kube-api-access-wjgw2\") pod \"perf-node-gather-daemonset-4nbqg\" (UID: \"a089c387-9c45-4841-9a1e-7bbf0328ab29\") " pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.281144 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.281122    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.552119 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.552098    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"]
Sep 17 23:37:52.554673 ip-10-0-140-43 kubenswrapper[2582]: W0917 23:37:52.554646    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda089c387_9c45_4841_9a1e_7bbf0328ab29.slice/crio-cffe759be206ac0c37356c7549482749c8d8435489782a1cd3fb13f94c8913e5 WatchSource:0}: Error finding container cffe759be206ac0c37356c7549482749c8d8435489782a1cd3fb13f94c8913e5: Status 404 returned error can't find the container with id cffe759be206ac0c37356c7549482749c8d8435489782a1cd3fb13f94c8913e5
Sep 17 23:37:52.829774 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.829703    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-hcp48_712b2a11-e159-4115-836d-9270325d32e0/volume-data-source-validator/0.log"
Sep 17 23:37:52.934275 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.934251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg" event={"ID":"a089c387-9c45-4841-9a1e-7bbf0328ab29","Type":"ContainerStarted","Data":"e2ac014b0b41f158f5dcdb8629bf988401d1c3dc5258277c0ab9082632a27d13"}
Sep 17 23:37:52.934413 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.934296    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg" event={"ID":"a089c387-9c45-4841-9a1e-7bbf0328ab29","Type":"ContainerStarted","Data":"cffe759be206ac0c37356c7549482749c8d8435489782a1cd3fb13f94c8913e5"}
Sep 17 23:37:52.934413 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.934387    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:52.952720 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:52.952679    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg" podStartSLOduration=1.95266593 podStartE2EDuration="1.95266593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:37:52.951045307 +0000 UTC m=+1627.638117884" watchObservedRunningTime="2026-09-17 23:37:52.95266593 +0000 UTC m=+1627.639738506"
Sep 17 23:37:53.899322 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:53.899291    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-msg6l_a14c19c4-990c-4ec5-996c-bbaca8aafc93/dns/0.log"
Sep 17 23:37:53.922967 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:53.922947    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-msg6l_a14c19c4-990c-4ec5-996c-bbaca8aafc93/kube-rbac-proxy/0.log"
Sep 17 23:37:54.025990 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:54.025965    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-95nk9_d0fe30d8-d67c-4736-86fe-e33d93dafbb0/dns-node-resolver/0.log"
Sep 17 23:37:54.491522 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:54.491491    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5999d9d4c4-8hg92_c9abe976-07fe-4a06-be0e-530da46ae4eb/registry/0.log"
Sep 17 23:37:54.541428 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:54.541405    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pxhs2_70ea03b1-9946-46d6-b68a-9c730d932c72/node-ca/0.log"
Sep 17 23:37:55.833055 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:55.833024    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-st4l9_27fd2159-4f7f-477e-89f6-9fb97acf078a/serve-healthcheck-canary/0.log"
Sep 17 23:37:56.290587 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:56.290558    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-wvxp9_2043c82f-26d1-4f3b-add6-385c35468db3/insights-operator/0.log"
Sep 17 23:37:56.291143 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:56.291124    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-wvxp9_2043c82f-26d1-4f3b-add6-385c35468db3/insights-operator/1.log"
Sep 17 23:37:56.314688 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:56.314656    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f2sxs_023d5e17-4668-4320-9949-9ec7f5a3feef/kube-rbac-proxy/0.log"
Sep 17 23:37:56.349466 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:56.349448    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f2sxs_023d5e17-4668-4320-9949-9ec7f5a3feef/exporter/0.log"
Sep 17 23:37:56.388476 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:56.388456    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f2sxs_023d5e17-4668-4320-9949-9ec7f5a3feef/extractor/0.log"
Sep 17 23:37:58.948459 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:58.948429    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wk5sv/perf-node-gather-daemonset-4nbqg"
Sep 17 23:37:59.063745 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:59.063717    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-bk44s_f8a6a9bd-6479-47cc-bb74-c081b63949a9/server/0.log"
Sep 17 23:37:59.258307 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:59.258230    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-qh6x4_a7a8953b-aa7d-43e3-8823-4e581c5f716d/manager/0.log"
Sep 17 23:37:59.350241 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:37:59.350211    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-9wdxd_f344da7b-9c68-4d6f-910c-cc71effd4fe7/seaweedfs/0.log"
Sep 17 23:38:04.203474 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:04.203438    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-6786h_54abc83b-5baf-48fa-b91f-72d90b85aca6/kube-storage-version-migrator-operator/0.log"
Sep 17 23:38:05.410021 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:05.409994    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j6vfx_720f72ab-3b57-4871-9d92-4f546cf4217f/kube-multus-additional-cni-plugins/0.log"
Sep 17 23:38:05.442279 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:05.442241    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j6vfx_720f72ab-3b57-4871-9d92-4f546cf4217f/egress-router-binary-copy/0.log"
Sep 17 23:38:05.485042 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:05.485005    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j6vfx_720f72ab-3b57-4871-9d92-4f546cf4217f/cni-plugins/0.log"
Sep 17 23:38:05.521221 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:05.521200    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j6vfx_720f72ab-3b57-4871-9d92-4f546cf4217f/bond-cni-plugin/0.log"
Sep 17 23:38:05.554884 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:05.554862    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j6vfx_720f72ab-3b57-4871-9d92-4f546cf4217f/routeoverride-cni/0.log"
Sep 17 23:38:05.582901 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:05.582878    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j6vfx_720f72ab-3b57-4871-9d92-4f546cf4217f/whereabouts-cni-bincopy/0.log"
Sep 17 23:38:05.606125 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:05.606101    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j6vfx_720f72ab-3b57-4871-9d92-4f546cf4217f/whereabouts-cni/0.log"
Sep 17 23:38:05.923639 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:05.923611    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-t2x98_473f1449-5a91-400d-8225-88e8716cd98d/kube-multus/0.log"
Sep 17 23:38:06.019155 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:06.019131    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nmn9p_80efd7c7-aa08-47ee-a917-251ef8624493/network-metrics-daemon/0.log"
Sep 17 23:38:06.047828 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:06.047811    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nmn9p_80efd7c7-aa08-47ee-a917-251ef8624493/kube-rbac-proxy/0.log"
Sep 17 23:38:06.922827 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:06.922795    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88d87_70ddc5e6-3ef1-4732-a37e-3fa6d0a84670/ovn-controller/0.log"
Sep 17 23:38:06.951740 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:06.951719    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88d87_70ddc5e6-3ef1-4732-a37e-3fa6d0a84670/ovn-acl-logging/0.log"
Sep 17 23:38:06.985013 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:06.984992    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88d87_70ddc5e6-3ef1-4732-a37e-3fa6d0a84670/kube-rbac-proxy-node/0.log"
Sep 17 23:38:07.018043 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:07.018024    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88d87_70ddc5e6-3ef1-4732-a37e-3fa6d0a84670/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 23:38:07.048690 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:07.048667    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88d87_70ddc5e6-3ef1-4732-a37e-3fa6d0a84670/northd/0.log"
Sep 17 23:38:07.092741 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:07.092725    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88d87_70ddc5e6-3ef1-4732-a37e-3fa6d0a84670/nbdb/0.log"
Sep 17 23:38:07.130992 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:07.130972    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88d87_70ddc5e6-3ef1-4732-a37e-3fa6d0a84670/sbdb/0.log"
Sep 17 23:38:07.202209 ip-10-0-140-43 kubenswrapper[2582]: I0917 23:38:07.202146    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88d87_70ddc5e6-3ef1-4732-a37e-3fa6d0a84670/ovnkube-controller/0.log"