Sep 17 19:06:34.377790 ip-10-0-128-177 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 19:06:38.215167 ip-10-0-128-177 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:38.215167 ip-10-0-128-177 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 19:06:38.215167 ip-10-0-128-177 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:38.215167 ip-10-0-128-177 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 19:06:38.215167 ip-10-0-128-177 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:38.216019 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.215247    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 19:06:38.218924 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218908    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:38.218924 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218923    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218927    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218932    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218935    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218938    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218941    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218945    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218948    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218950    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218953    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218956    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218958    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218961    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218964    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218966    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218969    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218971    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218974    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218976    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218979    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:38.218993 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218982    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218990    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218993    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218996    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.218999    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219002    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219004    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219007    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219010    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219013    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219016    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219018    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219021    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219023    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219026    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219029    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219031    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219034    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219036    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219040    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:38.219482 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219042    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219045    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219048    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219050    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219053    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219055    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219058    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219066    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219068    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219071    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219073    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219076    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219078    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219081    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219084    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219087    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219090    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219092    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219095    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219098    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:38.219975 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219100    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219103    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219106    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219108    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219111    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219113    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219116    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219119    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219121    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219124    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219126    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219129    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219132    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219134    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219137    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219140    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219144    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219148    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219150    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:38.220472 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219153    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219156    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219158    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219161    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219165    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219168    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219172    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219176    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219180    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219184    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219186    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219189    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219192    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219195    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219197    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219200    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219203    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219207    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219210    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:38.220926 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219960    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219967    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219970    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219974    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219977    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219980    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219983    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219985    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219989    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219992    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219995    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.219997    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220000    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220003    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220006    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220008    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220011    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220013    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220016    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220018    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:38.221387 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220022    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220025    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220029    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220033    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220036    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220039    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220042    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220045    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220047    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220050    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220053    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220055    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220058    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220060    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220063    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220066    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220068    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220071    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220073    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:38.221887 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220076    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220078    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220081    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220085    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220088    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220091    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220094    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220097    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220099    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220102    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220104    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220107    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220110    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220113    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220116    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220119    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220122    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220124    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220127    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:38.222360 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220129    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220134    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220137    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220140    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220143    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220146    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220148    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220151    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220155    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220158    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220160    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220163    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220165    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220168    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220170    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220174    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220177    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220181    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220184    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:38.222856 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220186    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220189    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220192    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220194    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220197    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220199    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220202    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220205    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220208    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220210    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220213    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220216    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220218    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220221    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220223    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220226    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220228    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220231    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220234    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220236    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:38.223316 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220239    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220241    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220602    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220607    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220610    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220613    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220616    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220619    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220621    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220624    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220628    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220632    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220635    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220638    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220641    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220643    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220646    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220649    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220653    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220656    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:38.223826 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220659    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220662    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220665    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220667    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220671    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220675    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220679    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220682    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220685    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220688    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220691    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220694    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220696    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220699    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220701    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220704    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220706    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220710    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220713    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:38.224352 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220715    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220718    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220720    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220723    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220726    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220729    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220731    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220734    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220737    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220739    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220742    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220746    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220749    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220752    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220755    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220757    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220760    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220762    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220765    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220767    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:38.224843 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220770    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220772    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220775    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220777    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220780    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220782    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220785    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220787    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220790    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220793    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220795    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220798    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220800    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220803    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220805    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220808    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220811    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220814    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220817    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220820    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:38.225348 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220823    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220826    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220828    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220832    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220836    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220839    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220842    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220845    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220848    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220851    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220853    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220856    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220859    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220861    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220864    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220866    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220869    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220872    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220874    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:38.225835 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220876    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220879    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.220881    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.220965    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221009    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221017    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221021    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221025    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221030    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221034    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221039    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221043    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221046    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221050    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221053    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221056    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221060    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221063    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221066    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221069    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221072    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221075    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221078    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 19:06:38.226301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221081    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221085    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221088    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221092    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221095    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221098    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221102    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221105    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221107    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221110    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221114    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221117    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221120    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221123    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221126    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221129    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221132    2583 flags.go:64] FLAG: --enable-server="true"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221137    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221141    2583 flags.go:64] FLAG: --event-burst="100"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221144    2583 flags.go:64] FLAG: --event-qps="50"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221147    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221151    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221154    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221158    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 19:06:38.226866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221160    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221164    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221168    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221171    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221174    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221177    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221180    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221183    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221186    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221189    2583 flags.go:64] FLAG: --feature-gates=""
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221193    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221196    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221200    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221203    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221206    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221209    2583 flags.go:64] FLAG: --help="false"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221212    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221215    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221218    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221221    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221224    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221228    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221231    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 19:06:38.227456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221234    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221237    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221242    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221244    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221248    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221250    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221254    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221260    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221263    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221266    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221269    2583 flags.go:64] FLAG: --lock-file=""
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221272    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221275    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221278    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221284    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221287    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221290    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221293    2583 flags.go:64] FLAG: --logging-format="text"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221295    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221299    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221302    2583 flags.go:64] FLAG: --manifest-url=""
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221305    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221309    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221312    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221316    2583 flags.go:64] FLAG: --max-pods="110"
Sep 17 19:06:38.228437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221319    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221322    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221325    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221329    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221332    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221335    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221338    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221345    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221348    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221353    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221356    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221359    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221364    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221367    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221371    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221377    2583 flags.go:64] FLAG: --port="10250"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221380    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221383    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a49715c3ed3e6e1c"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221386    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221389    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221392    2583 flags.go:64] FLAG: --register-node="true"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221395    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221399    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221402    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 17 19:06:38.229595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221405    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221408    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221411    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221414    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221417    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221420    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221423    2583 flags.go:64] FLAG: --runonce="false"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221426    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221429    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221432    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221434    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221448    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221452    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221455    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221458    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221461    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221464    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221468    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221471    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221474    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221477    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221480    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221485    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221488    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221491    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 19:06:38.230685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221496    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221500    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221503    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221506    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221509    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221512    2583 flags.go:64] FLAG: --v="2"
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221516    2583 flags.go:64] FLAG: --version="false"
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221520    2583 flags.go:64] FLAG: --vmodule=""
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221524    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221527    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221643    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221646    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221649    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221651    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221654    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221656    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221659    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221662    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221664    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221667    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221670    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221672    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:38.231631 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221675    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221677    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221681    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221684    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221686    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221689    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221692    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221695    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221697    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221700    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221703    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221707    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221710    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221713    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221716    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221719    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221721    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221724    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221726    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221729    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:38.232426 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221732    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221734    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221737    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221739    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221742    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221745    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221747    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221751    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221754    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221757    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221760    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221762    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221765    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221767    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221771    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221774    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221776    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221779    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221782    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221784    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:38.233284 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221787    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221790    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221792    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221797    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221799    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221802    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221805    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221807    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221810    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221812    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221815    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221818    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221821    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221823    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221826    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221829    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221831    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221834    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221851    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:38.233858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221855    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221858    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221861    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221864    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221867    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221870    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221872    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221877    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221880    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221884    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221887    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221891    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221895    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221898    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221901    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221904    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221910    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221913    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221916    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:38.234471 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221919    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:38.235009 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221922    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:38.235009 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221924    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:38.235009 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221927    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:38.235009 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221930    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:38.235009 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221932    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:38.235009 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221935    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:38.235009 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221938    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:38.235009 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.221940    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:38.235009 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.221945    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:38.235009 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.229760    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 19:06:38.235009 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.229778    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 19:06:38.235009 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.229807    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 19:06:38.235009 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.229817    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 19:06:38.235009 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.229923    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:38.235009 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.229932    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.229937    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.229941    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.229946    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.229950    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.229955    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.229962    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.229968    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.229973    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.229978    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.229983    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.229988    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.229992    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.229997    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230002    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230006    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230011    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230016    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230021    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:38.235428 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230025    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230030    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230034    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230039    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230043    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230048    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230052    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230057    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230061    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230067    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230071    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230076    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230080    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230084    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230088    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230093    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230097    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230101    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230105    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230110    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:38.235997 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230114    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230118    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230123    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230127    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230131    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230136    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230140    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230144    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230149    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230153    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230157    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230164    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230179    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230187    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230191    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230196    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230200    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230205    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230209    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:38.236536 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230213    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230218    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230222    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230228    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230232    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230236    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230240    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230245    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230249    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230253    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230258    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230262    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230267    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230271    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230275    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230279    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230283    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230288    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230292    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230296    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:38.236992 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230300    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230304    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230309    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230313    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230317    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230321    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230325    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230330    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230334    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230338    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230343    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230347    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230351    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230355    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230361    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230365    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230370    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230374    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230378    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:38.237807 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230382    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:38.238531 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.230390    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:38.238531 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230586    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:38.238531 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230595    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:38.238531 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230600    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:38.238531 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230604    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:38.238531 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230608    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:38.238531 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230613    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:38.238531 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230618    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:38.238531 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230623    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:38.238531 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230627    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:38.238531 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230632    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:38.238531 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230636    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:38.238531 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230640    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:38.238531 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230647    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230654    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230659    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230663    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230668    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230672    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230677    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230681    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230685    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230688    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230692    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230697    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230701    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230705    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230709    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230714    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230719    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230724    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230729    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230734    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:38.238903 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230738    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230743    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230747    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230752    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230756    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230760    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230764    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230769    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230773    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230777    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230782    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230786    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230790    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230795    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230799    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230803    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230807    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230812    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230816    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230820    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:38.239450 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230825    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230829    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230833    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230837    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230841    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230846    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230850    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230855    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230859    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230864    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230869    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230873    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230877    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230881    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230885    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230889    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230893    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230897    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230904    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230910    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:38.240063 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230914    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230918    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230922    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230927    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230930    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230935    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230939    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230943    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230947    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230951    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230956    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230960    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230964    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230968    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230972    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230976    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230980    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230985    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230989    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230994    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:38.240920 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.230998    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:38.241657 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.231002    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:38.241657 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.231007    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:38.241657 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.231011    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:38.241657 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.231015    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:38.241657 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.231019    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:38.241657 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:38.231025    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:38.241657 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.231034    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:38.241657 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.231228    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 19:06:38.241657 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.233915    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 19:06:38.241657 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.234736    2583 server.go:1019] "Starting client certificate rotation"
Sep 17 19:06:38.241657 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.234839    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 19:06:38.241657 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.234874    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 19:06:38.241657 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.238619    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 19:06:38.242034 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.241048    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 19:06:38.246142 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.246119    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 17 19:06:38.247782 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.247767    2583 log.go:25] "Validated CRI v1 image API"
Sep 17 19:06:38.250628 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.250613    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 19:06:38.254512 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.254486    2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7a4e1911-0d3e-42d8-946b-839f17d00bc7:/dev/nvme0n1p3 be9f97f5-8e20-4146-8c16-0edd0deb22b7:/dev/nvme0n1p4]
Sep 17 19:06:38.254594 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.254511    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 19:06:38.259501 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.259368    2583 manager.go:216] Machine: {Timestamp:2026-09-17 19:06:38.258907732 +0000 UTC m=+2.086765962 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097587 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c82d06b514b7829464a4658a568c8 SystemUUID:ec2c82d0-6b51-4b78-2946-4a4658a568c8 BootID:dda7ce80-ba21-4b32-a4b5-919253f6bf6f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:eb:b3:6c:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:eb:b3:6c:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:a9:24:9a:79:12 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 19:06:38.259501 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.259493    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 19:06:38.259656 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.259640    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 19:06:38.261275 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.261250    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 19:06:38.261412 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.261278    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-177.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 19:06:38.262633 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.262623    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 19:06:38.262672 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.262636    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 19:06:38.262672 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.262644    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 19:06:38.262672 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.262654    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 19:06:38.262780 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.262739    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 19:06:38.266508 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.266494    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 19:06:38.266617 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.266608    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 19:06:38.266656 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.266639    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 19:06:38.266656 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.266650    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 19:06:38.266736 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.266661    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 19:06:38.266736 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.266670    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 17 19:06:38.266736 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.266678    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 19:06:38.267117 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.267105    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 19:06:38.267205 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.267191    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 19:06:38.270424 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.268071    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 19:06:38.271389 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.271357    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 19:06:38.273650 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.273634    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 19:06:38.273736 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.273673    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 19:06:38.274859 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.274846    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 19:06:38.274943 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.274867    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 19:06:38.274943 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.274876    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 19:06:38.274943 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.274885    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 19:06:38.274943 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.274897    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 19:06:38.274943 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.274905    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 19:06:38.274943 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.274925    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 19:06:38.274943 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.274934    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 19:06:38.274943 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.274943    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 19:06:38.275262 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.274952    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 19:06:38.275262 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.274964    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 19:06:38.275262 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.274972    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 19:06:38.275262 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.274999    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 19:06:38.275262 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.275007    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 19:06:38.276456 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.276409    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 19:06:38.276959 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.276940    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 19:06:38.278832 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.278814    2583 server.go:1268] "Started kubelet"
Sep 17 19:06:38.278915 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.278865    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 19:06:38.278970 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.278918    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 19:06:38.279008 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.278995    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 19:06:38.279737 ip-10-0-128-177 systemd[1]: Started Kubernetes Kubelet.
Sep 17 19:06:38.280275 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.280136    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 19:06:38.280821 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.280731    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 19:06:38.280874 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.280827    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-177.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 19:06:38.284718 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.284692    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 19:06:38.286670 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.286658    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 19:06:38.286760 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.286743    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 19:06:38.286834 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.286752    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:38.287320 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.287305    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 19:06:38.287320 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.287308    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 19:06:38.287495 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.287331    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 19:06:38.287495 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.287470    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 19:06:38.287495 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.287479    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 19:06:38.287638 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.287540    2583 factory.go:55] Registering systemd factory
Sep 17 19:06:38.287638 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.287556    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 17 19:06:38.287750 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.287734    2583 factory.go:153] Registering CRI-O factory
Sep 17 19:06:38.287750 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.287750    2583 factory.go:223] Registration of the crio container factory successfully
Sep 17 19:06:38.287869 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.287797    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 19:06:38.287869 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.287811    2583 factory.go:103] Registering Raw factory
Sep 17 19:06:38.287869 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.287822    2583 manager.go:1196] Started watching for new ooms in manager
Sep 17 19:06:38.288152 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.288140    2583 manager.go:349] Starting recovery of all containers
Sep 17 19:06:38.289671 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.289646    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 19:06:38.313188 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.313157    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bxmbx"
Sep 17 19:06:38.313318 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.313305    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bxmbx"
Sep 17 19:06:38.317944 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.317925    2583 manager.go:354] Recovery completed
Sep 17 19:06:38.319980 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.319961    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 19:06:38.320707 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.319844    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-177.ec2.internal.18d6313349dee1db  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-177.ec2.internal,UID:ip-10-0-128-177.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-177.ec2.internal,},FirstTimestamp:2026-09-17 19:06:38.278787547 +0000 UTC m=+2.106645773,LastTimestamp:2026-09-17 19:06:38.278787547 +0000 UTC m=+2.106645773,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-177.ec2.internal,}"
Sep 17 19:06:38.321910 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.321896    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:38.323628 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.323614    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:38.323710 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.323646    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:38.323710 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.323660    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:38.324247 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.324233    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 19:06:38.324247 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.324245    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 19:06:38.324330 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.324260    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 19:06:38.324330 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.324260    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:38.325042 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.325026    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:38.325096 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.325054    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:38.325096 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.325063    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:38.325424 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.325413    2583 policy_none.go:49] "None policy: Start"
Sep 17 19:06:38.325480 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.325429    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 19:06:38.325480 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.325452    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 19:06:38.326512 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.326501    2583 policy_none.go:47] "Start"
Sep 17 19:06:38.371055 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.371038    2583 manager.go:337] "Starting Device Plugin manager"
Sep 17 19:06:38.382519 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.371074    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 19:06:38.382519 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.371084    2583 server.go:87] "Starting device plugin registration server"
Sep 17 19:06:38.382519 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.371346    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 19:06:38.382519 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.371375    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 19:06:38.382519 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.371464    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 19:06:38.382519 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.371538    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 19:06:38.382519 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.371547    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 19:06:38.382519 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.372080    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 19:06:38.382519 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.372119    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:38.444811 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.444781    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 19:06:38.445918 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.445886    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 19:06:38.446027 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.445949    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 19:06:38.446027 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.445981    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 19:06:38.446116 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.446025    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 19:06:38.448509 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.448490    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:38.472002 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.471948    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:38.472850 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.472831    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:38.472932 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.472863    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:38.472932 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.472874    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:38.472932 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.472897    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.483189 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.483176    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.483241 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.483196    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-177.ec2.internal\": node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:38.503685 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.503670    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:38.547030 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.546991    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-177.ec2.internal"]
Sep 17 19:06:38.547147 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.547103    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:38.550000 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.549983    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:38.550083 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.550017    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:38.550083 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.550032    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:38.552436 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.552423    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:38.552565 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.552552    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.552602 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.552588    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:38.553322 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.553306    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:38.553450 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.553335    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:38.553450 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.553348    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:38.553450 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.553363    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:38.553450 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.553387    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:38.553450 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.553399    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:38.555491 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.555475    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.555574 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.555506    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:38.556337 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.556321    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:38.556399 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.556354    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:38.556399 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.556366    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:38.572027 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.572007    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-177.ec2.internal\" not found" node="ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.575579 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.575563    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-177.ec2.internal\" not found" node="ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.589088 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.589057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b18bf748c3cf6a280e206fecb6969819-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal\" (UID: \"b18bf748c3cf6a280e206fecb6969819\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.589088 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.589087    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b18bf748c3cf6a280e206fecb6969819-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal\" (UID: \"b18bf748c3cf6a280e206fecb6969819\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.589227 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.589104    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ee8c51dd833e290a1526e3b75d4bd28-config\") pod \"kube-apiserver-proxy-ip-10-0-128-177.ec2.internal\" (UID: \"6ee8c51dd833e290a1526e3b75d4bd28\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.604537 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.604513    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:38.689321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.689284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b18bf748c3cf6a280e206fecb6969819-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal\" (UID: \"b18bf748c3cf6a280e206fecb6969819\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.689506 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.689329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b18bf748c3cf6a280e206fecb6969819-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal\" (UID: \"b18bf748c3cf6a280e206fecb6969819\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.689506 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.689351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ee8c51dd833e290a1526e3b75d4bd28-config\") pod \"kube-apiserver-proxy-ip-10-0-128-177.ec2.internal\" (UID: \"6ee8c51dd833e290a1526e3b75d4bd28\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.689506 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.689367    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b18bf748c3cf6a280e206fecb6969819-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal\" (UID: \"b18bf748c3cf6a280e206fecb6969819\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.689506 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.689379    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ee8c51dd833e290a1526e3b75d4bd28-config\") pod \"kube-apiserver-proxy-ip-10-0-128-177.ec2.internal\" (UID: \"6ee8c51dd833e290a1526e3b75d4bd28\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.689506 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.689403    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b18bf748c3cf6a280e206fecb6969819-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal\" (UID: \"b18bf748c3cf6a280e206fecb6969819\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.705396 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.705376    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:38.805847 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.805783    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:38.875166 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.875137    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.878819 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:38.878793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-177.ec2.internal"
Sep 17 19:06:38.906171 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:38.906139    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:39.006698 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:39.006647    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:39.107302 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:39.107213    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:39.207720 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:39.207685    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:39.234874 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:39.234842    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 19:06:39.235318 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:39.234977    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 19:06:39.256056 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:39.255899    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:39.308801 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:39.308773    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:39.318933 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:39.318906    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 19:01:38 +0000 UTC" deadline="2028-04-28 06:02:05.101812839 +0000 UTC"
Sep 17 19:06:39.319003 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:39.318933    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14122h55m25.782883439s"
Sep 17 19:06:39.409626 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:39.409557    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:39.463101 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:39.463062    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18bf748c3cf6a280e206fecb6969819.slice/crio-21be5e7bf0eba95cfd0f4a48d1755cde47a4b5bf13e044a674c879b5d43ea6ba WatchSource:0}: Error finding container 21be5e7bf0eba95cfd0f4a48d1755cde47a4b5bf13e044a674c879b5d43ea6ba: Status 404 returned error can't find the container with id 21be5e7bf0eba95cfd0f4a48d1755cde47a4b5bf13e044a674c879b5d43ea6ba
Sep 17 19:06:39.463555 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:39.463537    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee8c51dd833e290a1526e3b75d4bd28.slice/crio-9c0ddc9b7605a497a0b4dda2ad6c6e2fc36a48e4f543bf943d204668930ece77 WatchSource:0}: Error finding container 9c0ddc9b7605a497a0b4dda2ad6c6e2fc36a48e4f543bf943d204668930ece77: Status 404 returned error can't find the container with id 9c0ddc9b7605a497a0b4dda2ad6c6e2fc36a48e4f543bf943d204668930ece77
Sep 17 19:06:39.469434 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:39.469417    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:06:39.494764 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:39.494745    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:39.509967 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:39.509951    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:39.610547 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:39.610511    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:39.711021 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:39.710953    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-177.ec2.internal\" not found"
Sep 17 19:06:39.743812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:39.743783    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:39.787466 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:39.787418    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal"
Sep 17 19:06:39.803070 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:39.803049    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 19:06:39.804201 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:39.804177    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-177.ec2.internal"
Sep 17 19:06:39.813728 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:39.813696    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 19:06:40.267840 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.267782    2583 apiserver.go:52] "Watching apiserver"
Sep 17 19:06:40.276891 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.276871    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 19:06:40.277263 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.277243    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-ggdrv","openshift-cluster-node-tuning-operator/tuned-dbvvq","openshift-image-registry/node-ca-m68t4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal","openshift-multus/multus-additional-cni-plugins-j74f8","openshift-multus/multus-jq2zg","openshift-multus/network-metrics-daemon-r2lm8","openshift-ovn-kubernetes/ovnkube-node-9lk9z","kube-system/global-pull-secret-syncer-x5kmn","kube-system/kube-apiserver-proxy-ip-10-0-128-177.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8","openshift-dns/node-resolver-kdh4s","openshift-network-diagnostics/network-check-target-sv444","openshift-network-operator/iptables-alerter-csq8c"]
Sep 17 19:06:40.284401 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.284382    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:40.284558 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:40.284531    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:06:40.286622 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.286604    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.287663 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.287624    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 19:06:40.288906 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.288886    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m68t4"
Sep 17 19:06:40.289912 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.289890    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:40.290303 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.290272    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:40.290472 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.290457    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pb2tw\""
Sep 17 19:06:40.290738 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.290578    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:40.291171 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.291143    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 19:06:40.291265 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.291208    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 19:06:40.291323 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.291307    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2mj9k\""
Sep 17 19:06:40.292623 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.292604    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ggdrv"
Sep 17 19:06:40.292996 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.292977    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 19:06:40.295045 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.295027    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.295936 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.295920    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-snxf2\""
Sep 17 19:06:40.296038 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.295956    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 19:06:40.297300 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.297270    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.297837 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.297822    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 19:06:40.299061 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299044    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 19:06:40.299164 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-modprobe-d\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.299164 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299136    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-sysconfig\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.299278 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-systemd\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.299278 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-run\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.299278 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299236    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-tmp\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.299278 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299255    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w2v6\" (UniqueName: \"kubernetes.io/projected/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-kube-api-access-4w2v6\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.299494 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-kubernetes\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.299494 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-sysctl-conf\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.299494 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299336    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 19:06:40.299494 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299341    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-lib-modules\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.299494 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299350    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 19:06:40.299494 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299363    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-var-lib-kubelet\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.299494 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299400    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 19:06:40.299494 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299423    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-tuned\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.299494 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299463    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68a1d862-0402-4b78-a9fc-d61d8b1816ac-host\") pod \"node-ca-m68t4\" (UID: \"68a1d862-0402-4b78-a9fc-d61d8b1816ac\") " pod="openshift-image-registry/node-ca-m68t4"
Sep 17 19:06:40.299494 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299483    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-644x6\""
Sep 17 19:06:40.299899 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpvz5\" (UniqueName: \"kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5\") pod \"network-check-target-sv444\" (UID: \"f78a6d46-2733-4c1a-83a9-e566098efa57\") " pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:40.299899 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299563    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/68a1d862-0402-4b78-a9fc-d61d8b1816ac-serviceca\") pod \"node-ca-m68t4\" (UID: \"68a1d862-0402-4b78-a9fc-d61d8b1816ac\") " pod="openshift-image-registry/node-ca-m68t4"
Sep 17 19:06:40.299899 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh4fv\" (UniqueName: \"kubernetes.io/projected/68a1d862-0402-4b78-a9fc-d61d8b1816ac-kube-api-access-xh4fv\") pod \"node-ca-m68t4\" (UID: \"68a1d862-0402-4b78-a9fc-d61d8b1816ac\") " pod="openshift-image-registry/node-ca-m68t4"
Sep 17 19:06:40.299899 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299623    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-sysctl-d\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.299899 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299692    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-sys\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.299899 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299727    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 19:06:40.299899 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299727    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-host\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.299899 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.299883    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:40.300140 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:40.299947    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:06:40.301069 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.300692    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xg272\""
Sep 17 19:06:40.301482 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.301363    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 19:06:40.301568 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.301493    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 19:06:40.305104 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.305073    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.307458 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.307426    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:40.307542 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:40.307515    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:06:40.309346 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.309316    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 19:06:40.309453 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.309370    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 19:06:40.309781 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.309746    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-csq8c"
Sep 17 19:06:40.310348 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.310329    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 19:06:40.310522 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.310497    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 19:06:40.311408 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.311366    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 19:06:40.311514 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.311432    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9xlxm\""
Sep 17 19:06:40.311575 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.311453    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 19:06:40.312179 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.312160    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.314051 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.314034    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:40.314367 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.314349    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kdh4s"
Sep 17 19:06:40.315059 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.315008    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 19:06:40.319822 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.319752    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r469w\""
Sep 17 19:06:40.319914 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.319847    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:40.319914 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.319875    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 19:06:40.319994 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.319935    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 19:06:40.320062 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.320043    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 19:06:40.320062 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.320054    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7ldvk\""
Sep 17 19:06:40.320200 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.320093    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 19:06:40.320294 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.320277    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 19:06:40.320371 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.320277    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-87hmb\""
Sep 17 19:06:40.362642 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.362620    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2mvsb"
Sep 17 19:06:40.388626 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.388603    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 19:06:40.397701 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.397679    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2mvsb"
Sep 17 19:06:40.400459 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.400427    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-kubernetes\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.400554 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.400495    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h5dc\" (UniqueName: \"kubernetes.io/projected/bdbca049-57a2-46ed-bc0f-9dc645d58b24-kube-api-access-8h5dc\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.400554 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.400530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6cc77d6-c367-470c-bfca-7399dc10d65b-hosts-file\") pod \"node-resolver-kdh4s\" (UID: \"f6cc77d6-c367-470c-bfca-7399dc10d65b\") " pod="openshift-dns/node-resolver-kdh4s"
Sep 17 19:06:40.400657 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.400559    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-kubernetes\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.400657 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.400564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/68a1d862-0402-4b78-a9fc-d61d8b1816ac-serviceca\") pod \"node-ca-m68t4\" (UID: \"68a1d862-0402-4b78-a9fc-d61d8b1816ac\") " pod="openshift-image-registry/node-ca-m68t4"
Sep 17 19:06:40.400657 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.400598    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs\") pod \"network-metrics-daemon-r2lm8\" (UID: \"d7c98728-9f29-4061-9131-95535a68e5c0\") " pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:40.400657 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.400618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-run-systemd\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.400657 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.400632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-ovnkube-script-lib\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.400657 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.400650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-system-cni-dir\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.400935 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.400714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-run-k8s-cni-cncf-io\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.400935 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.400765    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-var-lib-kubelet\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.401409 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-sys\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.401505 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401405    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/68a1d862-0402-4b78-a9fc-d61d8b1816ac-serviceca\") pod \"node-ca-m68t4\" (UID: \"68a1d862-0402-4b78-a9fc-d61d8b1816ac\") " pod="openshift-image-registry/node-ca-m68t4"
Sep 17 19:06:40.401505 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401434    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nkff\" (UniqueName: \"kubernetes.io/projected/d7c98728-9f29-4061-9131-95535a68e5c0-kube-api-access-6nkff\") pod \"network-metrics-daemon-r2lm8\" (UID: \"d7c98728-9f29-4061-9131-95535a68e5c0\") " pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:40.401604 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-sys\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.401604 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401532    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-etc-selinux\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.401604 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401578    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fcs2\" (UniqueName: \"kubernetes.io/projected/f6cc77d6-c367-470c-bfca-7399dc10d65b-kube-api-access-6fcs2\") pod \"node-resolver-kdh4s\" (UID: \"f6cc77d6-c367-470c-bfca-7399dc10d65b\") " pod="openshift-dns/node-resolver-kdh4s"
Sep 17 19:06:40.401747 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-modprobe-d\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.401747 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401651    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-sysconfig\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.401747 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dee9f420-c08e-4466-80b6-dbb7d50d36ea-konnectivity-ca\") pod \"konnectivity-agent-ggdrv\" (UID: \"dee9f420-c08e-4466-80b6-dbb7d50d36ea\") " pod="kube-system/konnectivity-agent-ggdrv"
Sep 17 19:06:40.401747 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401717    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a1fa965-ba48-4ea1-993c-11ac62dd09b4-iptables-alerter-script\") pod \"iptables-alerter-csq8c\" (UID: \"9a1fa965-ba48-4ea1-993c-11ac62dd09b4\") " pod="openshift-network-operator/iptables-alerter-csq8c"
Sep 17 19:06:40.401925 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-etc-kubernetes\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.401925 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401772    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvsl4\" (UniqueName: \"kubernetes.io/projected/a41b66ba-345d-4b30-9987-043f67724e6e-kube-api-access-wvsl4\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.401925 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401795    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-modprobe-d\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.401925 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68a1d862-0402-4b78-a9fc-d61d8b1816ac-host\") pod \"node-ca-m68t4\" (UID: \"68a1d862-0402-4b78-a9fc-d61d8b1816ac\") " pod="openshift-image-registry/node-ca-m68t4"
Sep 17 19:06:40.401925 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401838    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.401925 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-sysconfig\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.402184 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.401931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-run-netns\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.402184 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402028    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68a1d862-0402-4b78-a9fc-d61d8b1816ac-host\") pod \"node-ca-m68t4\" (UID: \"68a1d862-0402-4b78-a9fc-d61d8b1816ac\") " pod="openshift-image-registry/node-ca-m68t4"
Sep 17 19:06:40.402184 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402077    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-run-multus-certs\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.402184 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402110    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6cc77d6-c367-470c-bfca-7399dc10d65b-tmp-dir\") pod \"node-resolver-kdh4s\" (UID: \"f6cc77d6-c367-470c-bfca-7399dc10d65b\") " pod="openshift-dns/node-resolver-kdh4s"
Sep 17 19:06:40.402184 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-etc-openvswitch\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.402402 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-multus-socket-dir-parent\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.402402 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402246    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-sysctl-d\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.402402 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-host\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.402402 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402338    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-systemd-units\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.402402 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402382    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-run-netns\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.402629 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402411    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-cni-bin\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.402629 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402437    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b47kq\" (UniqueName: \"kubernetes.io/projected/9a1fa965-ba48-4ea1-993c-11ac62dd09b4-kube-api-access-b47kq\") pod \"iptables-alerter-csq8c\" (UID: \"9a1fa965-ba48-4ea1-993c-11ac62dd09b4\") " pod="openshift-network-operator/iptables-alerter-csq8c"
Sep 17 19:06:40.402629 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402583    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-multus-cni-dir\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.402629 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402612    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-tmp\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.402831 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402655    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-ovn-node-metrics-cert\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.402831 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a41b66ba-345d-4b30-9987-043f67724e6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.402920 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-var-lib-kubelet\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.402967 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402925    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-tuned\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.402967 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402957    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-ovnkube-config\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.403057 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.402987    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z2tn\" (UniqueName: \"kubernetes.io/projected/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-kube-api-access-6z2tn\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.403057 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403017    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-var-lib-cni-bin\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.403057 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403040    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-device-dir\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.403188 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403088    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh4fv\" (UniqueName: \"kubernetes.io/projected/68a1d862-0402-4b78-a9fc-d61d8b1816ac-kube-api-access-xh4fv\") pod \"node-ca-m68t4\" (UID: \"68a1d862-0402-4b78-a9fc-d61d8b1816ac\") " pod="openshift-image-registry/node-ca-m68t4"
Sep 17 19:06:40.403188 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-slash\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.403188 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403173    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-log-socket\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.403316 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-cni-binary-copy\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.403316 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403242    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbjlg\" (UniqueName: \"kubernetes.io/projected/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-kube-api-access-kbjlg\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.403316 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403297    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-run-ovn\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.403463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403325    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-env-overrides\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.403463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a41b66ba-345d-4b30-9987-043f67724e6e-cnibin\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.403463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403373    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-hostroot\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.403762 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-socket-dir\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.403833 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403776    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-sys-fs\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.403833 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-run\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.403918 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403845    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w2v6\" (UniqueName: \"kubernetes.io/projected/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-kube-api-access-4w2v6\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.403966 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403916    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-cni-netd\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.404015 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.403974    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-var-lib-kubelet\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.404109 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.404090    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.404219 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.404146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-registration-dir\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.404311 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.404290    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-host\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.404400 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.404327    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-sysctl-d\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.404489 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.404467    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 19:06:40.404549 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.404509    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-run\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.404549 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.404479    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-sysctl-conf\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.404645 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.404594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-sysctl-conf\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.404645 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.404620    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-lib-modules\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.404764 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.404744    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dee9f420-c08e-4466-80b6-dbb7d50d36ea-agent-certs\") pod \"konnectivity-agent-ggdrv\" (UID: \"dee9f420-c08e-4466-80b6-dbb7d50d36ea\") " pod="kube-system/konnectivity-agent-ggdrv"
Sep 17 19:06:40.404813 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.404776    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-lib-modules\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.404906 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.404856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-var-lib-openvswitch\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.405009 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.404924    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-node-log\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.405009 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.404969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a41b66ba-345d-4b30-9987-043f67724e6e-system-cni-dir\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.405099 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvz5\" (UniqueName: \"kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5\") pod \"network-check-target-sv444\" (UID: \"f78a6d46-2733-4c1a-83a9-e566098efa57\") " pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:40.405211 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405184    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-kubelet\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.405303 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405234    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-run-ovn-kubernetes\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.405303 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a1fa965-ba48-4ea1-993c-11ac62dd09b4-host-slash\") pod \"iptables-alerter-csq8c\" (UID: \"9a1fa965-ba48-4ea1-993c-11ac62dd09b4\") " pod="openshift-network-operator/iptables-alerter-csq8c"
Sep 17 19:06:40.405402 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405313    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-cnibin\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.405402 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-var-lib-cni-multus\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.405505 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405411    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-multus-daemon-config\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.405505 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405463    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-run-openvswitch\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.405505 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405489    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a41b66ba-345d-4b30-9987-043f67724e6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.405626 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a41b66ba-345d-4b30-9987-043f67724e6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.405626 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405550    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f451007-41b2-4c22-8441-36a2b6284c60-kubelet-config\") pod \"global-pull-secret-syncer-x5kmn\" (UID: \"6f451007-41b2-4c22-8441-36a2b6284c60\") " pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:40.405626 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405578    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f451007-41b2-4c22-8441-36a2b6284c60-dbus\") pod \"global-pull-secret-syncer-x5kmn\" (UID: \"6f451007-41b2-4c22-8441-36a2b6284c60\") " pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:40.405626 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405599    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret\") pod \"global-pull-secret-syncer-x5kmn\" (UID: \"6f451007-41b2-4c22-8441-36a2b6284c60\") " pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:40.405816 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405630    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-systemd\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.405816 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a41b66ba-345d-4b30-9987-043f67724e6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.405816 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405725    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-multus-conf-dir\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.406018 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.405905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-systemd\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.407874 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.407853    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-tmp\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.407975 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.407953    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-etc-tuned\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.412656 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:40.412638    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:40.412724 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:40.412663    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:40.412724 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:40.412676    2583 projected.go:196] Error preparing data for projected volume kube-api-access-zpvz5 for pod openshift-network-diagnostics/network-check-target-sv444: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:40.412798 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:40.412750    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5 podName:f78a6d46-2733-4c1a-83a9-e566098efa57 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:40.912731167 +0000 UTC m=+4.740589394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zpvz5" (UniqueName: "kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5") pod "network-check-target-sv444" (UID: "f78a6d46-2733-4c1a-83a9-e566098efa57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:40.416312 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.416293    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w2v6\" (UniqueName: \"kubernetes.io/projected/5e7c41cc-0eda-45d2-8acb-b8f93a463a13-kube-api-access-4w2v6\") pod \"tuned-dbvvq\" (UID: \"5e7c41cc-0eda-45d2-8acb-b8f93a463a13\") " pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.420545 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.420527    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh4fv\" (UniqueName: \"kubernetes.io/projected/68a1d862-0402-4b78-a9fc-d61d8b1816ac-kube-api-access-xh4fv\") pod \"node-ca-m68t4\" (UID: \"68a1d862-0402-4b78-a9fc-d61d8b1816ac\") " pod="openshift-image-registry/node-ca-m68t4"
Sep 17 19:06:40.450867 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.450814    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-177.ec2.internal" event={"ID":"6ee8c51dd833e290a1526e3b75d4bd28","Type":"ContainerStarted","Data":"9c0ddc9b7605a497a0b4dda2ad6c6e2fc36a48e4f543bf943d204668930ece77"}
Sep 17 19:06:40.451957 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.451932    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal" event={"ID":"b18bf748c3cf6a280e206fecb6969819","Type":"ContainerStarted","Data":"21be5e7bf0eba95cfd0f4a48d1755cde47a4b5bf13e044a674c879b5d43ea6ba"}
Sep 17 19:06:40.506068 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506029    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvsl4\" (UniqueName: \"kubernetes.io/projected/a41b66ba-345d-4b30-9987-043f67724e6e-kube-api-access-wvsl4\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.506240 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506076    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.506240 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-run-netns\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.506240 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506124    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-run-multus-certs\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.506240 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506156    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6cc77d6-c367-470c-bfca-7399dc10d65b-tmp-dir\") pod \"node-resolver-kdh4s\" (UID: \"f6cc77d6-c367-470c-bfca-7399dc10d65b\") " pod="openshift-dns/node-resolver-kdh4s"
Sep 17 19:06:40.506240 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506175    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.506240 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506187    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-run-netns\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.506240 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506206    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-etc-openvswitch\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.506240 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506232    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-run-multus-certs\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506245    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-multus-socket-dir-parent\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506286    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-etc-openvswitch\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506302    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-multus-socket-dir-parent\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-systemd-units\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506335    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-systemd-units\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506341    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-run-netns\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506362    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-run-netns\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506366    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-cni-bin\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b47kq\" (UniqueName: \"kubernetes.io/projected/9a1fa965-ba48-4ea1-993c-11ac62dd09b4-kube-api-access-b47kq\") pod \"iptables-alerter-csq8c\" (UID: \"9a1fa965-ba48-4ea1-993c-11ac62dd09b4\") " pod="openshift-network-operator/iptables-alerter-csq8c"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506412    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-cni-bin\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-multus-cni-dir\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-ovn-node-metrics-cert\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a41b66ba-345d-4b30-9987-043f67724e6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506532    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-ovnkube-config\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506557    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z2tn\" (UniqueName: \"kubernetes.io/projected/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-kube-api-access-6z2tn\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506574    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6cc77d6-c367-470c-bfca-7399dc10d65b-tmp-dir\") pod \"node-resolver-kdh4s\" (UID: \"f6cc77d6-c367-470c-bfca-7399dc10d65b\") " pod="openshift-dns/node-resolver-kdh4s"
Sep 17 19:06:40.506676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506576    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-multus-cni-dir\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-var-lib-cni-bin\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-device-dir\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506680    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-var-lib-cni-bin\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506698    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-device-dir\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506782    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-slash\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506809    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-log-socket\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506831    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-slash\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-cni-binary-copy\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506891    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-log-socket\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506903    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbjlg\" (UniqueName: \"kubernetes.io/projected/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-kube-api-access-kbjlg\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-run-ovn\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.506967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-env-overrides\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a41b66ba-345d-4b30-9987-043f67724e6e-cnibin\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507025    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-hostroot\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507046    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-socket-dir\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-sys-fs\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-cni-netd\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.507490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507105    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a41b66ba-345d-4b30-9987-043f67724e6e-cnibin\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507119    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507142    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-registration-dir\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507155    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-run-ovn\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dee9f420-c08e-4466-80b6-dbb7d50d36ea-agent-certs\") pod \"konnectivity-agent-ggdrv\" (UID: \"dee9f420-c08e-4466-80b6-dbb7d50d36ea\") " pod="kube-system/konnectivity-agent-ggdrv"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507188    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a41b66ba-345d-4b30-9987-043f67724e6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507194    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-ovnkube-config\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507204    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-var-lib-openvswitch\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-node-log\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507269    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a41b66ba-345d-4b30-9987-043f67724e6e-system-cni-dir\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507287    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-cni-netd\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507338    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-sys-fs\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507343    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a41b66ba-345d-4b30-9987-043f67724e6e-system-cni-dir\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507344    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-registration-dir\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507384    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-node-log\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507396    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-var-lib-openvswitch\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507423    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-kubelet\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.508321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507431    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507431    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-cni-binary-copy\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-run-ovn-kubernetes\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a1fa965-ba48-4ea1-993c-11ac62dd09b4-host-slash\") pod \"iptables-alerter-csq8c\" (UID: \"9a1fa965-ba48-4ea1-993c-11ac62dd09b4\") " pod="openshift-network-operator/iptables-alerter-csq8c"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507502    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-run-ovn-kubernetes\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507508    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-hostroot\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-cnibin\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507524    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-env-overrides\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-socket-dir\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507542    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-host-kubelet\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507559    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-cnibin\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507581    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a1fa965-ba48-4ea1-993c-11ac62dd09b4-host-slash\") pod \"iptables-alerter-csq8c\" (UID: \"9a1fa965-ba48-4ea1-993c-11ac62dd09b4\") " pod="openshift-network-operator/iptables-alerter-csq8c"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-var-lib-cni-multus\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507657    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-multus-daemon-config\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507678    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-run-openvswitch\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507683    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-var-lib-cni-multus\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507716    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-run-openvswitch\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a41b66ba-345d-4b30-9987-043f67724e6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.509090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507821    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a41b66ba-345d-4b30-9987-043f67724e6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f451007-41b2-4c22-8441-36a2b6284c60-kubelet-config\") pod \"global-pull-secret-syncer-x5kmn\" (UID: \"6f451007-41b2-4c22-8441-36a2b6284c60\") " pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507888    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f451007-41b2-4c22-8441-36a2b6284c60-dbus\") pod \"global-pull-secret-syncer-x5kmn\" (UID: \"6f451007-41b2-4c22-8441-36a2b6284c60\") " pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507913    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret\") pod \"global-pull-secret-syncer-x5kmn\" (UID: \"6f451007-41b2-4c22-8441-36a2b6284c60\") " pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507946    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a41b66ba-345d-4b30-9987-043f67724e6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-multus-conf-dir\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.507996    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h5dc\" (UniqueName: \"kubernetes.io/projected/bdbca049-57a2-46ed-bc0f-9dc645d58b24-kube-api-access-8h5dc\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508019    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6cc77d6-c367-470c-bfca-7399dc10d65b-hosts-file\") pod \"node-resolver-kdh4s\" (UID: \"f6cc77d6-c367-470c-bfca-7399dc10d65b\") " pod="openshift-dns/node-resolver-kdh4s"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs\") pod \"network-metrics-daemon-r2lm8\" (UID: \"d7c98728-9f29-4061-9131-95535a68e5c0\") " pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508068    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-run-systemd\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-ovnkube-script-lib\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508115    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-system-cni-dir\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508141    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-run-k8s-cni-cncf-io\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508161    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-multus-daemon-config\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508219    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-var-lib-kubelet\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-var-lib-kubelet\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nkff\" (UniqueName: \"kubernetes.io/projected/d7c98728-9f29-4061-9131-95535a68e5c0-kube-api-access-6nkff\") pod \"network-metrics-daemon-r2lm8\" (UID: \"d7c98728-9f29-4061-9131-95535a68e5c0\") " pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:40.509909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-etc-selinux\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508301    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a41b66ba-345d-4b30-9987-043f67724e6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508313    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fcs2\" (UniqueName: \"kubernetes.io/projected/f6cc77d6-c367-470c-bfca-7399dc10d65b-kube-api-access-6fcs2\") pod \"node-resolver-kdh4s\" (UID: \"f6cc77d6-c367-470c-bfca-7399dc10d65b\") " pod="openshift-dns/node-resolver-kdh4s"
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dee9f420-c08e-4466-80b6-dbb7d50d36ea-konnectivity-ca\") pod \"konnectivity-agent-ggdrv\" (UID: \"dee9f420-c08e-4466-80b6-dbb7d50d36ea\") " pod="kube-system/konnectivity-agent-ggdrv"
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a1fa965-ba48-4ea1-993c-11ac62dd09b4-iptables-alerter-script\") pod \"iptables-alerter-csq8c\" (UID: \"9a1fa965-ba48-4ea1-993c-11ac62dd09b4\") " pod="openshift-network-operator/iptables-alerter-csq8c"
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508492    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-etc-kubernetes\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508592    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f451007-41b2-4c22-8441-36a2b6284c60-kubelet-config\") pod \"global-pull-secret-syncer-x5kmn\" (UID: \"6f451007-41b2-4c22-8441-36a2b6284c60\") " pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:40.508638    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508645    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f451007-41b2-4c22-8441-36a2b6284c60-dbus\") pod \"global-pull-secret-syncer-x5kmn\" (UID: \"6f451007-41b2-4c22-8441-36a2b6284c60\") " pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a41b66ba-345d-4b30-9987-043f67724e6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508681    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-etc-kubernetes\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:40.508729    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret podName:6f451007-41b2-4c22-8441-36a2b6284c60 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:41.008708973 +0000 UTC m=+4.836567186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret") pod "global-pull-secret-syncer-x5kmn" (UID: "6f451007-41b2-4c22-8441-36a2b6284c60") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:40.508758    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:40.508817    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs podName:d7c98728-9f29-4061-9131-95535a68e5c0 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:41.008801433 +0000 UTC m=+4.836659665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs") pod "network-metrics-daemon-r2lm8" (UID: "d7c98728-9f29-4061-9131-95535a68e5c0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508860    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-run-systemd\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508878    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a41b66ba-345d-4b30-9987-043f67724e6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508926    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-system-cni-dir\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.510463 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bdbca049-57a2-46ed-bc0f-9dc645d58b24-etc-selinux\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.511035 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.508988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-host-run-k8s-cni-cncf-io\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.511035 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.509028    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-multus-conf-dir\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.511035 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.509083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6cc77d6-c367-470c-bfca-7399dc10d65b-hosts-file\") pod \"node-resolver-kdh4s\" (UID: \"f6cc77d6-c367-470c-bfca-7399dc10d65b\") " pod="openshift-dns/node-resolver-kdh4s"
Sep 17 19:06:40.511035 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.509553    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9a1fa965-ba48-4ea1-993c-11ac62dd09b4-iptables-alerter-script\") pod \"iptables-alerter-csq8c\" (UID: \"9a1fa965-ba48-4ea1-993c-11ac62dd09b4\") " pod="openshift-network-operator/iptables-alerter-csq8c"
Sep 17 19:06:40.511035 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.509610    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-ovnkube-script-lib\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.511035 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.509724    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dee9f420-c08e-4466-80b6-dbb7d50d36ea-konnectivity-ca\") pod \"konnectivity-agent-ggdrv\" (UID: \"dee9f420-c08e-4466-80b6-dbb7d50d36ea\") " pod="kube-system/konnectivity-agent-ggdrv"
Sep 17 19:06:40.511035 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.509760    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-ovn-node-metrics-cert\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.511035 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.509857    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dee9f420-c08e-4466-80b6-dbb7d50d36ea-agent-certs\") pod \"konnectivity-agent-ggdrv\" (UID: \"dee9f420-c08e-4466-80b6-dbb7d50d36ea\") " pod="kube-system/konnectivity-agent-ggdrv"
Sep 17 19:06:40.530708 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.530655    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h5dc\" (UniqueName: \"kubernetes.io/projected/bdbca049-57a2-46ed-bc0f-9dc645d58b24-kube-api-access-8h5dc\") pod \"aws-ebs-csi-driver-node-6krj8\" (UID: \"bdbca049-57a2-46ed-bc0f-9dc645d58b24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.530878 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.530857    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbjlg\" (UniqueName: \"kubernetes.io/projected/3c6753d8-f30c-4345-9c1b-e0aec1d11c24-kube-api-access-kbjlg\") pod \"multus-jq2zg\" (UID: \"3c6753d8-f30c-4345-9c1b-e0aec1d11c24\") " pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.530922 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.530858    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvsl4\" (UniqueName: \"kubernetes.io/projected/a41b66ba-345d-4b30-9987-043f67724e6e-kube-api-access-wvsl4\") pod \"multus-additional-cni-plugins-j74f8\" (UID: \"a41b66ba-345d-4b30-9987-043f67724e6e\") " pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.531894 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.531858    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nkff\" (UniqueName: \"kubernetes.io/projected/d7c98728-9f29-4061-9131-95535a68e5c0-kube-api-access-6nkff\") pod \"network-metrics-daemon-r2lm8\" (UID: \"d7c98728-9f29-4061-9131-95535a68e5c0\") " pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:40.532553 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.532534    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z2tn\" (UniqueName: \"kubernetes.io/projected/f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c-kube-api-access-6z2tn\") pod \"ovnkube-node-9lk9z\" (UID: \"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c\") " pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.534016 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.533998    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b47kq\" (UniqueName: \"kubernetes.io/projected/9a1fa965-ba48-4ea1-993c-11ac62dd09b4-kube-api-access-b47kq\") pod \"iptables-alerter-csq8c\" (UID: \"9a1fa965-ba48-4ea1-993c-11ac62dd09b4\") " pod="openshift-network-operator/iptables-alerter-csq8c"
Sep 17 19:06:40.535392 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.535374    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fcs2\" (UniqueName: \"kubernetes.io/projected/f6cc77d6-c367-470c-bfca-7399dc10d65b-kube-api-access-6fcs2\") pod \"node-resolver-kdh4s\" (UID: \"f6cc77d6-c367-470c-bfca-7399dc10d65b\") " pod="openshift-dns/node-resolver-kdh4s"
Sep 17 19:06:40.597408 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.597372    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dbvvq"
Sep 17 19:06:40.606688 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.606666    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m68t4"
Sep 17 19:06:40.614657 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.614640    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ggdrv"
Sep 17 19:06:40.620527 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.620512    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j74f8"
Sep 17 19:06:40.628284 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.628262    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jq2zg"
Sep 17 19:06:40.634151 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.634122    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:06:40.640873 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.640851    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-csq8c"
Sep 17 19:06:40.647181 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.647164    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8"
Sep 17 19:06:40.653138 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:40.653117    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kdh4s"
Sep 17 19:06:41.012253 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.012166    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvz5\" (UniqueName: \"kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5\") pod \"network-check-target-sv444\" (UID: \"f78a6d46-2733-4c1a-83a9-e566098efa57\") " pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:41.012253 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.012223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret\") pod \"global-pull-secret-syncer-x5kmn\" (UID: \"6f451007-41b2-4c22-8441-36a2b6284c60\") " pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:41.012502 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.012256    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs\") pod \"network-metrics-daemon-r2lm8\" (UID: \"d7c98728-9f29-4061-9131-95535a68e5c0\") " pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:41.012502 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:41.012329    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:41.012502 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:41.012355    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:41.012502 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:41.012369    2583 projected.go:196] Error preparing data for projected volume kube-api-access-zpvz5 for pod openshift-network-diagnostics/network-check-target-sv444: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:41.012502 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:41.012378    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:41.012502 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:41.012389    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:41.012502 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:41.012469    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret podName:6f451007-41b2-4c22-8441-36a2b6284c60 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:42.012423776 +0000 UTC m=+5.840282003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret") pod "global-pull-secret-syncer-x5kmn" (UID: "6f451007-41b2-4c22-8441-36a2b6284c60") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:41.012502 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:41.012497    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5 podName:f78a6d46-2733-4c1a-83a9-e566098efa57 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:42.01248448 +0000 UTC m=+5.840342701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpvz5" (UniqueName: "kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5") pod "network-check-target-sv444" (UID: "f78a6d46-2733-4c1a-83a9-e566098efa57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:41.012815 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:41.012520    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs podName:d7c98728-9f29-4061-9131-95535a68e5c0 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:42.012511026 +0000 UTC m=+5.840369240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs") pod "network-metrics-daemon-r2lm8" (UID: "d7c98728-9f29-4061-9131-95535a68e5c0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:41.266950 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:41.266908    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf75cfa47_2a6a_4335_aeb7_2f3fd0fa156c.slice/crio-91c6a72bdcfea470f6e28889767dca590e999617183a0afd6337635684b2f082 WatchSource:0}: Error finding container 91c6a72bdcfea470f6e28889767dca590e999617183a0afd6337635684b2f082: Status 404 returned error can't find the container with id 91c6a72bdcfea470f6e28889767dca590e999617183a0afd6337635684b2f082
Sep 17 19:06:41.267434 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:41.267412    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1fa965_ba48_4ea1_993c_11ac62dd09b4.slice/crio-450d4ee7bb13f7284962dd27085f75b607c4424c10f49a4c0a5ac230d033011c WatchSource:0}: Error finding container 450d4ee7bb13f7284962dd27085f75b607c4424c10f49a4c0a5ac230d033011c: Status 404 returned error can't find the container with id 450d4ee7bb13f7284962dd27085f75b607c4424c10f49a4c0a5ac230d033011c
Sep 17 19:06:41.268566 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:41.268517    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7c41cc_0eda_45d2_8acb_b8f93a463a13.slice/crio-255c2e0fb73e18b6aee81cd968fcf844f410ff9a270ab2e85609c95816513b05 WatchSource:0}: Error finding container 255c2e0fb73e18b6aee81cd968fcf844f410ff9a270ab2e85609c95816513b05: Status 404 returned error can't find the container with id 255c2e0fb73e18b6aee81cd968fcf844f410ff9a270ab2e85609c95816513b05
Sep 17 19:06:41.269611 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:41.269589    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6cc77d6_c367_470c_bfca_7399dc10d65b.slice/crio-d389a60bf70ab49e9eef177c672bbcb2dc980edb5de9daf61e2dfe6f0906ea3f WatchSource:0}: Error finding container d389a60bf70ab49e9eef177c672bbcb2dc980edb5de9daf61e2dfe6f0906ea3f: Status 404 returned error can't find the container with id d389a60bf70ab49e9eef177c672bbcb2dc980edb5de9daf61e2dfe6f0906ea3f
Sep 17 19:06:41.272355 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:41.272252    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6753d8_f30c_4345_9c1b_e0aec1d11c24.slice/crio-6307fc23ddc2737accc4c604bec3afabad5d806fefde76e5309fb85bd5029fa0 WatchSource:0}: Error finding container 6307fc23ddc2737accc4c604bec3afabad5d806fefde76e5309fb85bd5029fa0: Status 404 returned error can't find the container with id 6307fc23ddc2737accc4c604bec3afabad5d806fefde76e5309fb85bd5029fa0
Sep 17 19:06:41.274364 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:41.274345    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41b66ba_345d_4b30_9987_043f67724e6e.slice/crio-dbe5313c162525b4324797574221de11d09d7b378b764036c8c7fd394d00a758 WatchSource:0}: Error finding container dbe5313c162525b4324797574221de11d09d7b378b764036c8c7fd394d00a758: Status 404 returned error can't find the container with id dbe5313c162525b4324797574221de11d09d7b378b764036c8c7fd394d00a758
Sep 17 19:06:41.275623 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:41.275519    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbca049_57a2_46ed_bc0f_9dc645d58b24.slice/crio-d9850e732dbcb10ae13501291d5df8edb2be694b0ec8134e6e5c3f31f0a8a1ad WatchSource:0}: Error finding container d9850e732dbcb10ae13501291d5df8edb2be694b0ec8134e6e5c3f31f0a8a1ad: Status 404 returned error can't find the container with id d9850e732dbcb10ae13501291d5df8edb2be694b0ec8134e6e5c3f31f0a8a1ad
Sep 17 19:06:41.284967 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:06:41.284949    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a1d862_0402_4b78_a9fc_d61d8b1816ac.slice/crio-db8b93c0c06723e85448404925c1edf90c274266ec02f05a3e12d17eb32d641a WatchSource:0}: Error finding container db8b93c0c06723e85448404925c1edf90c274266ec02f05a3e12d17eb32d641a: Status 404 returned error can't find the container with id db8b93c0c06723e85448404925c1edf90c274266ec02f05a3e12d17eb32d641a
Sep 17 19:06:41.401317 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.401247    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 19:01:40 +0000 UTC" deadline="2028-05-30 13:35:00.764719926 +0000 UTC"
Sep 17 19:06:41.401317 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.401312    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14898h28m19.363411386s"
Sep 17 19:06:41.458106 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.458058    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-177.ec2.internal" event={"ID":"6ee8c51dd833e290a1526e3b75d4bd28","Type":"ContainerStarted","Data":"d4dda8a1fcc601be39e2c3af18b47219d1fa5895890c122e277370acdf70f988"}
Sep 17 19:06:41.460245 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.460221    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ggdrv" event={"ID":"dee9f420-c08e-4466-80b6-dbb7d50d36ea","Type":"ContainerStarted","Data":"7f92d03a8dc4caa534cbea7ac5dbd01a98891dd429e23e82cab8f8d851c585d0"}
Sep 17 19:06:41.461220 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.461194    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8" event={"ID":"bdbca049-57a2-46ed-bc0f-9dc645d58b24","Type":"ContainerStarted","Data":"d9850e732dbcb10ae13501291d5df8edb2be694b0ec8134e6e5c3f31f0a8a1ad"}
Sep 17 19:06:41.462489 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.462468    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j74f8" event={"ID":"a41b66ba-345d-4b30-9987-043f67724e6e","Type":"ContainerStarted","Data":"dbe5313c162525b4324797574221de11d09d7b378b764036c8c7fd394d00a758"}
Sep 17 19:06:41.464902 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.464878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jq2zg" event={"ID":"3c6753d8-f30c-4345-9c1b-e0aec1d11c24","Type":"ContainerStarted","Data":"6307fc23ddc2737accc4c604bec3afabad5d806fefde76e5309fb85bd5029fa0"}
Sep 17 19:06:41.467208 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.467188    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z" event={"ID":"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c","Type":"ContainerStarted","Data":"91c6a72bdcfea470f6e28889767dca590e999617183a0afd6337635684b2f082"}
Sep 17 19:06:41.468700 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.468680    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m68t4" event={"ID":"68a1d862-0402-4b78-a9fc-d61d8b1816ac","Type":"ContainerStarted","Data":"db8b93c0c06723e85448404925c1edf90c274266ec02f05a3e12d17eb32d641a"}
Sep 17 19:06:41.470451 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.470409    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-csq8c" event={"ID":"9a1fa965-ba48-4ea1-993c-11ac62dd09b4","Type":"ContainerStarted","Data":"450d4ee7bb13f7284962dd27085f75b607c4424c10f49a4c0a5ac230d033011c"}
Sep 17 19:06:41.472712 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.472689    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kdh4s" event={"ID":"f6cc77d6-c367-470c-bfca-7399dc10d65b","Type":"ContainerStarted","Data":"d389a60bf70ab49e9eef177c672bbcb2dc980edb5de9daf61e2dfe6f0906ea3f"}
Sep 17 19:06:41.473889 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.473869    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dbvvq" event={"ID":"5e7c41cc-0eda-45d2-8acb-b8f93a463a13","Type":"ContainerStarted","Data":"255c2e0fb73e18b6aee81cd968fcf844f410ff9a270ab2e85609c95816513b05"}
Sep 17 19:06:41.474100 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:41.474060    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-177.ec2.internal" podStartSLOduration=2.474046548 podStartE2EDuration="2.474046548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:41.473640929 +0000 UTC m=+5.301499183" watchObservedRunningTime="2026-09-17 19:06:41.474046548 +0000 UTC m=+5.301904784"
Sep 17 19:06:42.019516 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:42.019419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvz5\" (UniqueName: \"kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5\") pod \"network-check-target-sv444\" (UID: \"f78a6d46-2733-4c1a-83a9-e566098efa57\") " pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:42.019516 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:42.019497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret\") pod \"global-pull-secret-syncer-x5kmn\" (UID: \"6f451007-41b2-4c22-8441-36a2b6284c60\") " pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:42.019729 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:42.019532    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs\") pod \"network-metrics-daemon-r2lm8\" (UID: \"d7c98728-9f29-4061-9131-95535a68e5c0\") " pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:42.019729 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:42.019688    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:42.019828 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:42.019731    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:42.019828 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:42.019749    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs podName:d7c98728-9f29-4061-9131-95535a68e5c0 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:44.019730194 +0000 UTC m=+7.847588415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs") pod "network-metrics-daemon-r2lm8" (UID: "d7c98728-9f29-4061-9131-95535a68e5c0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:42.019828 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:42.019760    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:42.019828 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:42.019774    2583 projected.go:196] Error preparing data for projected volume kube-api-access-zpvz5 for pod openshift-network-diagnostics/network-check-target-sv444: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:42.020043 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:42.019829    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5 podName:f78a6d46-2733-4c1a-83a9-e566098efa57 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:44.019809445 +0000 UTC m=+7.847667681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpvz5" (UniqueName: "kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5") pod "network-check-target-sv444" (UID: "f78a6d46-2733-4c1a-83a9-e566098efa57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:42.020043 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:42.019897    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:42.020043 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:42.019929    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret podName:6f451007-41b2-4c22-8441-36a2b6284c60 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:44.01991877 +0000 UTC m=+7.847776984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret") pod "global-pull-secret-syncer-x5kmn" (UID: "6f451007-41b2-4c22-8441-36a2b6284c60") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:42.402413 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:42.402311    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 19:01:40 +0000 UTC" deadline="2028-04-18 18:15:13.3412655 +0000 UTC"
Sep 17 19:06:42.402413 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:42.402350    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13895h8m30.938921066s"
Sep 17 19:06:42.450618 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:42.449476    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:42.450618 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:42.449821    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:06:42.450618 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:42.450282    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:42.450618 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:42.450380    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:06:42.450618 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:42.450464    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:42.450618 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:42.450549    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:06:42.496798 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:42.496761    2583 generic.go:356] "Generic (PLEG): container finished" podID="b18bf748c3cf6a280e206fecb6969819" containerID="1e92f7359bccf3426edb0b0dc37208e3cfae33399d9f3cb32024b2f7ddcc4050" exitCode=0
Sep 17 19:06:42.496966 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:42.496863    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal" event={"ID":"b18bf748c3cf6a280e206fecb6969819","Type":"ContainerDied","Data":"1e92f7359bccf3426edb0b0dc37208e3cfae33399d9f3cb32024b2f7ddcc4050"}
Sep 17 19:06:43.512112 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:43.512071    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal" event={"ID":"b18bf748c3cf6a280e206fecb6969819","Type":"ContainerStarted","Data":"5bb0c0fc4a97a96f229108f81ebb8613241af940c69f35f5773d85045c3b8f7c"}
Sep 17 19:06:44.037390 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:44.035858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvz5\" (UniqueName: \"kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5\") pod \"network-check-target-sv444\" (UID: \"f78a6d46-2733-4c1a-83a9-e566098efa57\") " pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:44.037390 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:44.035961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret\") pod \"global-pull-secret-syncer-x5kmn\" (UID: \"6f451007-41b2-4c22-8441-36a2b6284c60\") " pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:44.037390 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:44.035995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs\") pod \"network-metrics-daemon-r2lm8\" (UID: \"d7c98728-9f29-4061-9131-95535a68e5c0\") " pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:44.037390 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:44.036151    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:44.037390 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:44.036229    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs podName:d7c98728-9f29-4061-9131-95535a68e5c0 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:48.036194366 +0000 UTC m=+11.864052601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs") pod "network-metrics-daemon-r2lm8" (UID: "d7c98728-9f29-4061-9131-95535a68e5c0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:44.037390 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:44.036734    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:44.037390 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:44.036758    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:44.037390 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:44.036771    2583 projected.go:196] Error preparing data for projected volume kube-api-access-zpvz5 for pod openshift-network-diagnostics/network-check-target-sv444: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:44.037390 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:44.036817    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5 podName:f78a6d46-2733-4c1a-83a9-e566098efa57 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:48.036802095 +0000 UTC m=+11.864660323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpvz5" (UniqueName: "kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5") pod "network-check-target-sv444" (UID: "f78a6d46-2733-4c1a-83a9-e566098efa57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:44.037390 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:44.036874    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:44.037390 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:44.036909    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret podName:6f451007-41b2-4c22-8441-36a2b6284c60 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:48.036897232 +0000 UTC m=+11.864755445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret") pod "global-pull-secret-syncer-x5kmn" (UID: "6f451007-41b2-4c22-8441-36a2b6284c60") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:44.446797 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:44.446657    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:44.446797 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:44.446671    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:44.447021 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:44.446796    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:06:44.447021 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:44.446805    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:44.447021 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:44.446904    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:06:44.447021 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:44.446993    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:06:46.446789 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:46.446756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:46.446789 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:46.446779    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:46.447504 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:46.446838    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:46.447504 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:46.446947    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:06:46.447504 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:46.447337    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:06:46.447504 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:46.447425    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:06:48.070668 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:48.070629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvz5\" (UniqueName: \"kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5\") pod \"network-check-target-sv444\" (UID: \"f78a6d46-2733-4c1a-83a9-e566098efa57\") " pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:48.071124 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:48.070700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret\") pod \"global-pull-secret-syncer-x5kmn\" (UID: \"6f451007-41b2-4c22-8441-36a2b6284c60\") " pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:48.071124 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:48.070732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs\") pod \"network-metrics-daemon-r2lm8\" (UID: \"d7c98728-9f29-4061-9131-95535a68e5c0\") " pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:48.071124 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:48.070780    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:48.071124 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:48.070798    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:48.071124 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:48.070812    2583 projected.go:196] Error preparing data for projected volume kube-api-access-zpvz5 for pod openshift-network-diagnostics/network-check-target-sv444: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:48.071124 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:48.070828    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:48.071124 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:48.070866    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5 podName:f78a6d46-2733-4c1a-83a9-e566098efa57 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:56.070849197 +0000 UTC m=+19.898707430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpvz5" (UniqueName: "kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5") pod "network-check-target-sv444" (UID: "f78a6d46-2733-4c1a-83a9-e566098efa57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:48.071124 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:48.070885    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs podName:d7c98728-9f29-4061-9131-95535a68e5c0 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:56.070875228 +0000 UTC m=+19.898733446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs") pod "network-metrics-daemon-r2lm8" (UID: "d7c98728-9f29-4061-9131-95535a68e5c0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:48.071124 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:48.070893    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:48.071124 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:48.070923    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret podName:6f451007-41b2-4c22-8441-36a2b6284c60 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:56.0709126 +0000 UTC m=+19.898770828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret") pod "global-pull-secret-syncer-x5kmn" (UID: "6f451007-41b2-4c22-8441-36a2b6284c60") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:48.448925 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:48.447690    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:48.448925 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:48.447819    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:06:48.448925 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:48.448195    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:48.448925 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:48.448284    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:06:48.448925 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:48.448335    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:48.448925 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:48.448397    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:06:50.447193 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:50.447155    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:50.447773 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:50.447155    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:50.447773 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:50.447295    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:06:50.447773 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:50.447327    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:50.447773 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:50.447470    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:06:50.447773 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:50.447547    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:06:52.449588 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:52.449554    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:52.450045 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:52.449683    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:06:52.450118 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:52.450081    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:52.450200 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:52.450178    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:06:52.450245 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:52.450237    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:52.450310 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:52.450295    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:06:54.446692 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:54.446656    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:54.446692 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:54.446695    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:54.447206 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:54.446673    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:54.447206 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:54.446789    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:06:54.447206 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:54.446886    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:06:54.447206 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:54.446950    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:06:56.130529 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:56.130495    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvz5\" (UniqueName: \"kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5\") pod \"network-check-target-sv444\" (UID: \"f78a6d46-2733-4c1a-83a9-e566098efa57\") " pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:56.130992 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:56.130538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret\") pod \"global-pull-secret-syncer-x5kmn\" (UID: \"6f451007-41b2-4c22-8441-36a2b6284c60\") " pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:56.130992 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:56.130565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs\") pod \"network-metrics-daemon-r2lm8\" (UID: \"d7c98728-9f29-4061-9131-95535a68e5c0\") " pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:56.130992 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:56.130658    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:56.130992 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:56.130675    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:56.130992 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:56.130668    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:56.130992 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:56.130719    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret podName:6f451007-41b2-4c22-8441-36a2b6284c60 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:12.130704383 +0000 UTC m=+35.958562602 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret") pod "global-pull-secret-syncer-x5kmn" (UID: "6f451007-41b2-4c22-8441-36a2b6284c60") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:56.130992 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:56.130733    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:56.130992 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:56.130745    2583 projected.go:196] Error preparing data for projected volume kube-api-access-zpvz5 for pod openshift-network-diagnostics/network-check-target-sv444: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:56.130992 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:56.130736    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs podName:d7c98728-9f29-4061-9131-95535a68e5c0 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:12.130726911 +0000 UTC m=+35.958585124 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs") pod "network-metrics-daemon-r2lm8" (UID: "d7c98728-9f29-4061-9131-95535a68e5c0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:56.130992 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:56.130786    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5 podName:f78a6d46-2733-4c1a-83a9-e566098efa57 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:12.13077077 +0000 UTC m=+35.958628987 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zpvz5" (UniqueName: "kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5") pod "network-check-target-sv444" (UID: "f78a6d46-2733-4c1a-83a9-e566098efa57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:56.446316 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:56.446232    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:56.446316 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:56.446251    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:56.446556 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:56.446236    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:56.446556 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:56.446365    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:06:56.446556 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:56.446434    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:06:56.446556 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:56.446540    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:06:58.447490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:58.447424    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:06:58.447891 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:58.447550    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:06:58.447891 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:58.447595    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:06:58.447891 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:58.447693    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:06:58.447891 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:58.447751    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:06:58.447891 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:06:58.447834    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:06:58.542812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:58.542781    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ggdrv" event={"ID":"dee9f420-c08e-4466-80b6-dbb7d50d36ea","Type":"ContainerStarted","Data":"8b5888af3491e61a9e0df62b1e003fb65073c988c59372fdb25ff14dc50c44cf"}
Sep 17 19:06:58.544292 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:58.544263    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jq2zg" event={"ID":"3c6753d8-f30c-4345-9c1b-e0aec1d11c24","Type":"ContainerStarted","Data":"93de0830a8f63c1199aafaadd5032bdad6a992ae25d1db97cb717292693f8eb5"}
Sep 17 19:06:58.545733 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:58.545706    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m68t4" event={"ID":"68a1d862-0402-4b78-a9fc-d61d8b1816ac","Type":"ContainerStarted","Data":"1233100c17517d86096bc9ea0e11fe8c1fe451f42a90c40061ee0579562233f5"}
Sep 17 19:06:58.547146 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:58.547127    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dbvvq" event={"ID":"5e7c41cc-0eda-45d2-8acb-b8f93a463a13","Type":"ContainerStarted","Data":"65d14f0138235cb69588c5279e09394156438354e80661425b460daa96d5228f"}
Sep 17 19:06:58.562485 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:58.562417    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-177.ec2.internal" podStartSLOduration=19.562403454 podStartE2EDuration="19.562403454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:43.531037403 +0000 UTC m=+7.358895639" watchObservedRunningTime="2026-09-17 19:06:58.562403454 +0000 UTC m=+22.390261688"
Sep 17 19:06:58.586499 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:58.586320    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ggdrv" podStartSLOduration=3.5983981 podStartE2EDuration="20.586298504s" totalImagesPullingTime="16.987900404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:58.562755629 +0000 UTC m=+22.390613866" watchObservedRunningTime="2026-09-17 19:06:58.586298504 +0000 UTC m=+22.414156737"
Sep 17 19:06:58.589234 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:58.589188    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dbvvq" podStartSLOduration=3.517385544 podStartE2EDuration="20.589174746s" totalImagesPullingTime="17.071789202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:58.586765024 +0000 UTC m=+22.414623257" watchObservedRunningTime="2026-09-17 19:06:58.589174746 +0000 UTC m=+22.417032998"
Sep 17 19:06:58.622882 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:58.622827    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m68t4" podStartSLOduration=3.601673392 podStartE2EDuration="20.622801194s" totalImagesPullingTime="17.021127802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:58.622531287 +0000 UTC m=+22.450389521" watchObservedRunningTime="2026-09-17 19:06:58.622801194 +0000 UTC m=+22.450659431"
Sep 17 19:06:59.550523 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:59.550288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kdh4s" event={"ID":"f6cc77d6-c367-470c-bfca-7399dc10d65b","Type":"ContainerStarted","Data":"ac390d8e5bad7d1bd1a7406786184a3e2e492f9c969504865bc02f735da62297"}
Sep 17 19:06:59.551954 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:59.551932    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8" event={"ID":"bdbca049-57a2-46ed-bc0f-9dc645d58b24","Type":"ContainerStarted","Data":"eba332ecfc78f16288acab1599c92fdf46706bdb315c8e904cced732eadbb89a"}
Sep 17 19:06:59.553240 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:59.553218    2583 generic.go:356] "Generic (PLEG): container finished" podID="a41b66ba-345d-4b30-9987-043f67724e6e" containerID="ef96e5246fc9d13b8541c7dbdc4857638e38cd7142553bcb6a4c39763387c1f2" exitCode=0
Sep 17 19:06:59.553315 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:59.553295    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j74f8" event={"ID":"a41b66ba-345d-4b30-9987-043f67724e6e","Type":"ContainerDied","Data":"ef96e5246fc9d13b8541c7dbdc4857638e38cd7142553bcb6a4c39763387c1f2"}
Sep 17 19:06:59.557855 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:59.557837    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:06:59.558199 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:59.558183    2583 generic.go:356] "Generic (PLEG): container finished" podID="f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c" containerID="a4b8c1ae5b8e48ddc95d28fe106006f94c48e585d15d84f1aadf2cd06d0168c0" exitCode=1
Sep 17 19:06:59.558277 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:59.558261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z" event={"ID":"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c","Type":"ContainerStarted","Data":"97e7d03bf42ca23b7fb9b7dc2e48d2c70b491e1427b3820120fee425ca7e268f"}
Sep 17 19:06:59.558321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:59.558314    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z" event={"ID":"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c","Type":"ContainerStarted","Data":"18e5f03b16c212d6058472c5232648a1c7a2a26882bfd4571d456ba2f6ef7adc"}
Sep 17 19:06:59.558353 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:59.558346    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z" event={"ID":"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c","Type":"ContainerStarted","Data":"0e07699c8d5d32302fcf4d09199b4fa981d324c6cfaab7e4b49844664443c952"}
Sep 17 19:06:59.558387 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:59.558374    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z" event={"ID":"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c","Type":"ContainerStarted","Data":"5cd4f97a4a6955015e0a4f6672860b8c39b76a0a21b362ff5307da112cd7bea4"}
Sep 17 19:06:59.558423 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:59.558401    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z" event={"ID":"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c","Type":"ContainerDied","Data":"a4b8c1ae5b8e48ddc95d28fe106006f94c48e585d15d84f1aadf2cd06d0168c0"}
Sep 17 19:06:59.558482 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:59.558431    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z" event={"ID":"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c","Type":"ContainerStarted","Data":"9e315c4ebd0a1497eee92780e2a027a966ba0899253e7ac246aaacaa69b1b25a"}
Sep 17 19:06:59.571383 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:59.571351    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jq2zg" podStartSLOduration=4.498116471 podStartE2EDuration="21.571337957s" totalImagesPullingTime="17.073221486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:58.654556711 +0000 UTC m=+22.482414947" watchObservedRunningTime="2026-09-17 19:06:59.571337957 +0000 UTC m=+23.399196266"
Sep 17 19:06:59.571701 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:59.571680    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kdh4s" podStartSLOduration=4.495744153 podStartE2EDuration="21.571674646s" totalImagesPullingTime="17.075930493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:59.571179625 +0000 UTC m=+23.399037861" watchObservedRunningTime="2026-09-17 19:06:59.571674646 +0000 UTC m=+23.399532881"
Sep 17 19:06:59.926564 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:06:59.926529    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 19:07:00.384265 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:00.384147    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T19:06:59.926560325Z","UUID":"ff733299-a3e5-4294-b995-b1b49046a3d8","Handler":null,"Name":"","Endpoint":""}
Sep 17 19:07:00.385868 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:00.385828    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 19:07:00.385868 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:00.385861    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 19:07:00.446800 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:00.446725    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:07:00.446800 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:00.446742    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:07:00.446800 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:00.446735    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:07:00.447065 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:00.446871    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:07:00.447293 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:00.447200    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:07:00.447293 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:00.447259    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:07:00.562877 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:00.562830    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-csq8c" event={"ID":"9a1fa965-ba48-4ea1-993c-11ac62dd09b4","Type":"ContainerStarted","Data":"d8bbc36ab995f2b15f87451c60a7885d758107b3fde52e5008c45c1be8ff8fca"}
Sep 17 19:07:00.565489 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:00.565436    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8" event={"ID":"bdbca049-57a2-46ed-bc0f-9dc645d58b24","Type":"ContainerStarted","Data":"5cb0a3ee5b32b33c40246a47b6f4611152dbdf6ed3ede94e31b78879553b8f40"}
Sep 17 19:07:00.590555 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:00.590503    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-csq8c" podStartSLOduration=5.518993558 podStartE2EDuration="22.590483213s" totalImagesPullingTime="17.071489655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:00.590402532 +0000 UTC m=+24.418260768" watchObservedRunningTime="2026-09-17 19:07:00.590483213 +0000 UTC m=+24.418341449"
Sep 17 19:07:01.570453 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:01.570399    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8" event={"ID":"bdbca049-57a2-46ed-bc0f-9dc645d58b24","Type":"ContainerStarted","Data":"d9675ffc940cb7c2eba3a754edd742fa545fe893adfc768f7abb0684495104b4"}
Sep 17 19:07:01.573999 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:01.573976    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:07:01.574580 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:01.574554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z" event={"ID":"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c","Type":"ContainerStarted","Data":"b0802388045334f307df85cf6b8f1924bc14dbcfeab0ee72fd1eb20b5f339c66"}
Sep 17 19:07:01.591801 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:01.591764    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6krj8" podStartSLOduration=4.474120124 podStartE2EDuration="23.591750961s" totalImagesPullingTime="19.117630837s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:01.591310989 +0000 UTC m=+25.419169223" watchObservedRunningTime="2026-09-17 19:07:01.591750961 +0000 UTC m=+25.419609195"
Sep 17 19:07:01.757791 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:01.757750    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ggdrv"
Sep 17 19:07:01.758534 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:01.758512    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ggdrv"
Sep 17 19:07:02.446384 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:02.446299    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:07:02.446384 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:02.446317    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:07:02.446627 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:02.446432    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:07:02.446627 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:02.446495    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:07:02.446627 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:02.446556    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:07:02.446768 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:02.446665    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:07:03.582485 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:03.582310    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:07:03.582919 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:03.582871    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z" event={"ID":"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c","Type":"ContainerStarted","Data":"d1a6b72b89e4131a4dbd9e35e0dc79f13aae77bce005313b185c88b5c1571dc9"}
Sep 17 19:07:03.583280 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:03.583259    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:07:03.583397 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:03.583337    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:07:03.583397 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:03.583394    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:07:03.583515 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:03.583338    2583 scope.go:117] "RemoveContainer" containerID="a4b8c1ae5b8e48ddc95d28fe106006f94c48e585d15d84f1aadf2cd06d0168c0"
Sep 17 19:07:03.599112 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:03.599090    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:07:03.600801 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:03.600780    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:07:04.446544 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:04.446515    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:07:04.446676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:04.446547    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:07:04.446676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:04.446523    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:07:04.446676 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:04.446622    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:07:04.446793 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:04.446675    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:07:04.446793 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:04.446765    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:07:04.586114 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:04.586077    2583 generic.go:356] "Generic (PLEG): container finished" podID="a41b66ba-345d-4b30-9987-043f67724e6e" containerID="05f5a67881155772e034e58542526a7eaf0b5c1c3447fdbd6f04cd7b440fdae5" exitCode=0
Sep 17 19:07:04.586839 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:04.586158    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j74f8" event={"ID":"a41b66ba-345d-4b30-9987-043f67724e6e","Type":"ContainerDied","Data":"05f5a67881155772e034e58542526a7eaf0b5c1c3447fdbd6f04cd7b440fdae5"}
Sep 17 19:07:04.593113 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:04.593092    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:07:04.593502 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:04.593478    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z" event={"ID":"f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c","Type":"ContainerStarted","Data":"efcadc239b044bd41157217fab08f5612e1ff18b65e1e37db821f1ea5bea0ec7"}
Sep 17 19:07:04.643435 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:04.643401    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z" podStartSLOduration=9.289706293 podStartE2EDuration="26.64338985s" totalImagesPullingTime="17.353683557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:04.642283805 +0000 UTC m=+28.470142039" watchObservedRunningTime="2026-09-17 19:07:04.64338985 +0000 UTC m=+28.471248070"
Sep 17 19:07:05.504843 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:05.504577    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sv444"]
Sep 17 19:07:05.505041 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:05.504965    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:07:05.505136 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:05.505083    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:07:05.506009 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:05.505983    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x5kmn"]
Sep 17 19:07:05.506130 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:05.506089    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:07:05.506222 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:05.506199    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:07:05.506836 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:05.506817    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r2lm8"]
Sep 17 19:07:05.506923 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:05.506910    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:07:05.507001 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:05.506987    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:07:06.599811 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:06.599781    2583 generic.go:356] "Generic (PLEG): container finished" podID="a41b66ba-345d-4b30-9987-043f67724e6e" containerID="c304afa9395e4e56896b93944d3a552a4c090371ccf789f7cba18e17dbfe34f8" exitCode=0
Sep 17 19:07:06.600148 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:06.599831    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j74f8" event={"ID":"a41b66ba-345d-4b30-9987-043f67724e6e","Type":"ContainerDied","Data":"c304afa9395e4e56896b93944d3a552a4c090371ccf789f7cba18e17dbfe34f8"}
Sep 17 19:07:07.446736 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:07.446704    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:07:07.446736 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:07.446743    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:07:07.446916 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:07.446706    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:07:07.446916 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:07.446808    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:07:07.446979 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:07.446931    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:07:07.447032 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:07.447014    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:07:08.606595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:08.606558    2583 generic.go:356] "Generic (PLEG): container finished" podID="a41b66ba-345d-4b30-9987-043f67724e6e" containerID="2c8920ac72cba414dc24ea401ca43e0fe768d8f15d649961270b5841e535b792" exitCode=0
Sep 17 19:07:08.606595 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:08.606600    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j74f8" event={"ID":"a41b66ba-345d-4b30-9987-043f67724e6e","Type":"ContainerDied","Data":"2c8920ac72cba414dc24ea401ca43e0fe768d8f15d649961270b5841e535b792"}
Sep 17 19:07:09.446270 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:09.446239    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:07:09.446481 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:09.446245    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:07:09.446481 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:09.446387    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2lm8" podUID="d7c98728-9f29-4061-9131-95535a68e5c0"
Sep 17 19:07:09.446481 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:09.446245    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:07:09.446481 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:09.446426    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sv444" podUID="f78a6d46-2733-4c1a-83a9-e566098efa57"
Sep 17 19:07:09.446640 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:09.446513    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5kmn" podUID="6f451007-41b2-4c22-8441-36a2b6284c60"
Sep 17 19:07:10.086695 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:10.086661    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ggdrv"
Sep 17 19:07:10.087131 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:10.086855    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 19:07:10.087330 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:10.087293    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ggdrv"
Sep 17 19:07:11.264610 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.264528    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-177.ec2.internal" event="NodeReady"
Sep 17 19:07:11.265047 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.264682    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 19:07:11.307825 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.307785    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-648597d777-lfb4m"]
Sep 17 19:07:11.343006 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.342976    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-66n6k"]
Sep 17 19:07:11.343167 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.343116    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.347486 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.347155    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 19:07:11.347486 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.347175    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 19:07:11.347486 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.347192    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-msk9c\""
Sep 17 19:07:11.347486 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.347369    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 19:07:11.353312 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.352896    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 19:07:11.357116 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.357094    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-648597d777-lfb4m"]
Sep 17 19:07:11.357116 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.357127    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vtv4x"]
Sep 17 19:07:11.357278 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.357221    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:11.360272 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.360246    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 19:07:11.360369 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.360281    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dpmzt\""
Sep 17 19:07:11.360683 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.360661    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 19:07:11.377019 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.376988    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-66n6k"]
Sep 17 19:07:11.377172 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.377039    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vtv4x"]
Sep 17 19:07:11.377172 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.377135    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vtv4x"
Sep 17 19:07:11.381328 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.381162    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4xslp\""
Sep 17 19:07:11.381328 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.381170    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 19:07:11.382249 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.382216    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 19:07:11.382658 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.382634    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 19:07:11.438907 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.438869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gmxh\" (UniqueName: \"kubernetes.io/projected/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-kube-api-access-6gmxh\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:11.438907 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.438907    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74441563-19c2-4fe5-8802-9db0426c73f3-registry-certificates\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.439145 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.438941    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2vk2\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-kube-api-access-k2vk2\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.439145 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.438969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:11.439145 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.438994    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.439145 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.439118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-config-volume\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:11.439325 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.439168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74441563-19c2-4fe5-8802-9db0426c73f3-ca-trust-extracted\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.439325 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.439199    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74441563-19c2-4fe5-8802-9db0426c73f3-image-registry-private-configuration\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.439325 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.439224    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74441563-19c2-4fe5-8802-9db0426c73f3-trusted-ca\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.439325 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.439248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74441563-19c2-4fe5-8802-9db0426c73f3-installation-pull-secrets\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.439325 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.439272    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-bound-sa-token\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.439325 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.439303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-tmp-dir\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:11.447085 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.447054    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:07:11.447229 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.447067    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:07:11.447229 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.447067    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:07:11.450258 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.450232    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 19:07:11.450377 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.450270    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-47zq2\""
Sep 17 19:07:11.450377 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.450349    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b9ftw\""
Sep 17 19:07:11.450377 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.450349    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 19:07:11.450573 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.450527    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 19:07:11.450612 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.450571    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 19:07:11.539899 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.539865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-config-volume\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:11.540062 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.539922    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74441563-19c2-4fe5-8802-9db0426c73f3-ca-trust-extracted\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.540062 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.540045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74441563-19c2-4fe5-8802-9db0426c73f3-image-registry-private-configuration\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.540142 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.540084    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74441563-19c2-4fe5-8802-9db0426c73f3-trusted-ca\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.540142 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.540108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74441563-19c2-4fe5-8802-9db0426c73f3-installation-pull-secrets\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.540142 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.540129    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-bound-sa-token\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.540238 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.540163    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-tmp-dir\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:11.540238 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.540184    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gmxh\" (UniqueName: \"kubernetes.io/projected/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-kube-api-access-6gmxh\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:11.540238 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.540208    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74441563-19c2-4fe5-8802-9db0426c73f3-registry-certificates\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.540323 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.540241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2vk2\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-kube-api-access-k2vk2\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.540323 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.540267    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:11.540323 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.540290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.540420 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.540318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkxj7\" (UniqueName: \"kubernetes.io/projected/14e87624-a049-426c-b0ef-8d21d1100337-kube-api-access-nkxj7\") pod \"ingress-canary-vtv4x\" (UID: \"14e87624-a049-426c-b0ef-8d21d1100337\") " pod="openshift-ingress-canary/ingress-canary-vtv4x"
Sep 17 19:07:11.540420 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.540390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14e87624-a049-426c-b0ef-8d21d1100337-cert\") pod \"ingress-canary-vtv4x\" (UID: \"14e87624-a049-426c-b0ef-8d21d1100337\") " pod="openshift-ingress-canary/ingress-canary-vtv4x"
Sep 17 19:07:11.540511 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.540492    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74441563-19c2-4fe5-8802-9db0426c73f3-ca-trust-extracted\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.541130 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.540727    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-config-volume\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:11.541130 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:11.540832    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:07:11.541130 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:11.540848    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648597d777-lfb4m: secret "image-registry-tls" not found
Sep 17 19:07:11.541130 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:11.540912    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls podName:74441563-19c2-4fe5-8802-9db0426c73f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:12.04089649 +0000 UTC m=+35.868754703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls") pod "image-registry-648597d777-lfb4m" (UID: "74441563-19c2-4fe5-8802-9db0426c73f3") : secret "image-registry-tls" not found
Sep 17 19:07:11.541130 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.540964    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74441563-19c2-4fe5-8802-9db0426c73f3-registry-certificates\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.541130 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:11.541027    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:07:11.541130 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:11.541084    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls podName:92e4146f-3a14-40d3-bbb5-83bbe2f8be67 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:12.041070097 +0000 UTC m=+35.868928328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls") pod "dns-default-66n6k" (UID: "92e4146f-3a14-40d3-bbb5-83bbe2f8be67") : secret "dns-default-metrics-tls" not found
Sep 17 19:07:11.541130 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.541113    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-tmp-dir\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:11.541613 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.541553    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74441563-19c2-4fe5-8802-9db0426c73f3-trusted-ca\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.545594 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.545574    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74441563-19c2-4fe5-8802-9db0426c73f3-image-registry-private-configuration\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.545711 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.545575    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74441563-19c2-4fe5-8802-9db0426c73f3-installation-pull-secrets\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.550906 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.550880    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-bound-sa-token\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.551014 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.550884    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gmxh\" (UniqueName: \"kubernetes.io/projected/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-kube-api-access-6gmxh\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:11.551014 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.551002    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2vk2\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-kube-api-access-k2vk2\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:11.641360 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.641326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkxj7\" (UniqueName: \"kubernetes.io/projected/14e87624-a049-426c-b0ef-8d21d1100337-kube-api-access-nkxj7\") pod \"ingress-canary-vtv4x\" (UID: \"14e87624-a049-426c-b0ef-8d21d1100337\") " pod="openshift-ingress-canary/ingress-canary-vtv4x"
Sep 17 19:07:11.641531 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.641380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14e87624-a049-426c-b0ef-8d21d1100337-cert\") pod \"ingress-canary-vtv4x\" (UID: \"14e87624-a049-426c-b0ef-8d21d1100337\") " pod="openshift-ingress-canary/ingress-canary-vtv4x"
Sep 17 19:07:11.644203 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.644178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14e87624-a049-426c-b0ef-8d21d1100337-cert\") pod \"ingress-canary-vtv4x\" (UID: \"14e87624-a049-426c-b0ef-8d21d1100337\") " pod="openshift-ingress-canary/ingress-canary-vtv4x"
Sep 17 19:07:11.650860 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.650824    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkxj7\" (UniqueName: \"kubernetes.io/projected/14e87624-a049-426c-b0ef-8d21d1100337-kube-api-access-nkxj7\") pod \"ingress-canary-vtv4x\" (UID: \"14e87624-a049-426c-b0ef-8d21d1100337\") " pod="openshift-ingress-canary/ingress-canary-vtv4x"
Sep 17 19:07:11.690028 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:11.689991    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vtv4x"
Sep 17 19:07:12.044410 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:12.044372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:12.044657 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:12.044420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:12.044657 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:12.044498    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:07:12.044657 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:12.044559    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:07:12.044657 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:12.044575    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648597d777-lfb4m: secret "image-registry-tls" not found
Sep 17 19:07:12.044657 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:12.044577    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls podName:92e4146f-3a14-40d3-bbb5-83bbe2f8be67 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:13.044554945 +0000 UTC m=+36.872413160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls") pod "dns-default-66n6k" (UID: "92e4146f-3a14-40d3-bbb5-83bbe2f8be67") : secret "dns-default-metrics-tls" not found
Sep 17 19:07:12.044657 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:12.044626    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls podName:74441563-19c2-4fe5-8802-9db0426c73f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:13.044616341 +0000 UTC m=+36.872474573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls") pod "image-registry-648597d777-lfb4m" (UID: "74441563-19c2-4fe5-8802-9db0426c73f3") : secret "image-registry-tls" not found
Sep 17 19:07:12.145085 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:12.145044    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs\") pod \"network-metrics-daemon-r2lm8\" (UID: \"d7c98728-9f29-4061-9131-95535a68e5c0\") " pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:07:12.145261 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:12.145140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvz5\" (UniqueName: \"kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5\") pod \"network-check-target-sv444\" (UID: \"f78a6d46-2733-4c1a-83a9-e566098efa57\") " pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:07:12.145261 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:12.145167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret\") pod \"global-pull-secret-syncer-x5kmn\" (UID: \"6f451007-41b2-4c22-8441-36a2b6284c60\") " pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:07:12.145261 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:12.145235    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 19:07:12.145395 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:12.145311    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs podName:d7c98728-9f29-4061-9131-95535a68e5c0 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:44.145292531 +0000 UTC m=+67.973150749 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs") pod "network-metrics-daemon-r2lm8" (UID: "d7c98728-9f29-4061-9131-95535a68e5c0") : secret "metrics-daemon-secret" not found
Sep 17 19:07:12.147715 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:12.147693    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f451007-41b2-4c22-8441-36a2b6284c60-original-pull-secret\") pod \"global-pull-secret-syncer-x5kmn\" (UID: \"6f451007-41b2-4c22-8441-36a2b6284c60\") " pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:07:12.148012 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:12.147994    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpvz5\" (UniqueName: \"kubernetes.io/projected/f78a6d46-2733-4c1a-83a9-e566098efa57-kube-api-access-zpvz5\") pod \"network-check-target-sv444\" (UID: \"f78a6d46-2733-4c1a-83a9-e566098efa57\") " pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:07:12.358627 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:12.358543    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5kmn"
Sep 17 19:07:12.371834 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:12.371796    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:07:12.958661 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:12.958638    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sv444"]
Sep 17 19:07:12.963607 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:12.963578    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x5kmn"]
Sep 17 19:07:12.966681 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:12.966659    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vtv4x"]
Sep 17 19:07:12.972314 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:12.971660    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e87624_a049_426c_b0ef_8d21d1100337.slice/crio-2563bae8f19bd968cb8c8abd27a6d5004e697ba9c09dbfdb6aacdceeddc20b0d WatchSource:0}: Error finding container 2563bae8f19bd968cb8c8abd27a6d5004e697ba9c09dbfdb6aacdceeddc20b0d: Status 404 returned error can't find the container with id 2563bae8f19bd968cb8c8abd27a6d5004e697ba9c09dbfdb6aacdceeddc20b0d
Sep 17 19:07:12.972889 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:12.972848    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f451007_41b2_4c22_8441_36a2b6284c60.slice/crio-bd4fd5565e88cb0f1f088836522666976d1280381322e6dc0d1af585bb3e2edb WatchSource:0}: Error finding container bd4fd5565e88cb0f1f088836522666976d1280381322e6dc0d1af585bb3e2edb: Status 404 returned error can't find the container with id bd4fd5565e88cb0f1f088836522666976d1280381322e6dc0d1af585bb3e2edb
Sep 17 19:07:12.974053 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:12.973798    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78a6d46_2733_4c1a_83a9_e566098efa57.slice/crio-8d2468e6e34bceba2634a43b9d857424a57edb1876391b82104a5b7c948165b8 WatchSource:0}: Error finding container 8d2468e6e34bceba2634a43b9d857424a57edb1876391b82104a5b7c948165b8: Status 404 returned error can't find the container with id 8d2468e6e34bceba2634a43b9d857424a57edb1876391b82104a5b7c948165b8
Sep 17 19:07:13.052796 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.052576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:13.052902 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.052821    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:13.052902 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:13.052729    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:07:13.053011 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:13.052944    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls podName:92e4146f-3a14-40d3-bbb5-83bbe2f8be67 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:15.052924007 +0000 UTC m=+38.880782229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls") pod "dns-default-66n6k" (UID: "92e4146f-3a14-40d3-bbb5-83bbe2f8be67") : secret "dns-default-metrics-tls" not found
Sep 17 19:07:13.053011 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:13.053002    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:07:13.053115 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:13.053022    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648597d777-lfb4m: secret "image-registry-tls" not found
Sep 17 19:07:13.053115 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:13.053083    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls podName:74441563-19c2-4fe5-8802-9db0426c73f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:15.053066079 +0000 UTC m=+38.880924304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls") pod "image-registry-648597d777-lfb4m" (UID: "74441563-19c2-4fe5-8802-9db0426c73f3") : secret "image-registry-tls" not found
Sep 17 19:07:13.628133 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.628094    2583 generic.go:356] "Generic (PLEG): container finished" podID="a41b66ba-345d-4b30-9987-043f67724e6e" containerID="1354df38566397435836f2bbd9f15ec205cc18f8fbe8d94936c375a01882dbd9" exitCode=0
Sep 17 19:07:13.629071 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.628206    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j74f8" event={"ID":"a41b66ba-345d-4b30-9987-043f67724e6e","Type":"ContainerDied","Data":"1354df38566397435836f2bbd9f15ec205cc18f8fbe8d94936c375a01882dbd9"}
Sep 17 19:07:13.631505 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.631355    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x5kmn" event={"ID":"6f451007-41b2-4c22-8441-36a2b6284c60","Type":"ContainerStarted","Data":"bd4fd5565e88cb0f1f088836522666976d1280381322e6dc0d1af585bb3e2edb"}
Sep 17 19:07:13.634344 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.633675    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sv444" event={"ID":"f78a6d46-2733-4c1a-83a9-e566098efa57","Type":"ContainerStarted","Data":"8d2468e6e34bceba2634a43b9d857424a57edb1876391b82104a5b7c948165b8"}
Sep 17 19:07:13.636129 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.636059    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vtv4x" event={"ID":"14e87624-a049-426c-b0ef-8d21d1100337","Type":"ContainerStarted","Data":"2563bae8f19bd968cb8c8abd27a6d5004e697ba9c09dbfdb6aacdceeddc20b0d"}
Sep 17 19:07:13.831094 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.831019    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x4c96"]
Sep 17 19:07:13.833591 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.833570    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:13.838462 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.837631    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 19:07:13.838462 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.837903    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 19:07:13.838462 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.838197    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 19:07:13.838462 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.838300    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r9nrg\""
Sep 17 19:07:13.839248 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.839049    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 19:07:13.853526 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.853183    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x4c96"]
Sep 17 19:07:13.961320 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.961285    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b65a92c1-7983-4dc0-afd3-eed290a55ffb-data-volume\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:13.961529 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.961376    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b65a92c1-7983-4dc0-afd3-eed290a55ffb-crio-socket\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:13.961529 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.961408    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:13.961529 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.961433    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd5cj\" (UniqueName: \"kubernetes.io/projected/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-api-access-fd5cj\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:13.961529 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:13.961497    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:14.063411 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:14.062688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:14.063411 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:14.062763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b65a92c1-7983-4dc0-afd3-eed290a55ffb-data-volume\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:14.063411 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:14.062836    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b65a92c1-7983-4dc0-afd3-eed290a55ffb-crio-socket\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:14.063411 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:14.062861    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:14.063411 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:14.062887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd5cj\" (UniqueName: \"kubernetes.io/projected/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-api-access-fd5cj\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:14.063411 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:14.063267    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b65a92c1-7983-4dc0-afd3-eed290a55ffb-data-volume\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:14.063411 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:14.063293    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b65a92c1-7983-4dc0-afd3-eed290a55ffb-crio-socket\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:14.063411 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:14.063353    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:07:14.063411 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:14.063359    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:14.063411 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:14.063406    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls podName:b65a92c1-7983-4dc0-afd3-eed290a55ffb nodeName:}" failed. No retries permitted until 2026-09-17 19:07:14.563388262 +0000 UTC m=+38.391246490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4c96" (UID: "b65a92c1-7983-4dc0-afd3-eed290a55ffb") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:14.063411 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:14.063421    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm podName:b65a92c1-7983-4dc0-afd3-eed290a55ffb nodeName:}" failed. No retries permitted until 2026-09-17 19:07:14.563412764 +0000 UTC m=+38.391270977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x4c96" (UID: "b65a92c1-7983-4dc0-afd3-eed290a55ffb") : configmap "kube-rbac-proxy" not found
Sep 17 19:07:14.079745 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:14.079702    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd5cj\" (UniqueName: \"kubernetes.io/projected/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-api-access-fd5cj\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:14.568965 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:14.568933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:14.569164 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:14.569040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:14.569164 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:14.569105    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:14.569164 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:14.569113    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:07:14.569372 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:14.569177    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm podName:b65a92c1-7983-4dc0-afd3-eed290a55ffb nodeName:}" failed. No retries permitted until 2026-09-17 19:07:15.569158242 +0000 UTC m=+39.397016458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x4c96" (UID: "b65a92c1-7983-4dc0-afd3-eed290a55ffb") : configmap "kube-rbac-proxy" not found
Sep 17 19:07:14.569655 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:14.569637    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls podName:b65a92c1-7983-4dc0-afd3-eed290a55ffb nodeName:}" failed. No retries permitted until 2026-09-17 19:07:15.569614414 +0000 UTC m=+39.397472641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4c96" (UID: "b65a92c1-7983-4dc0-afd3-eed290a55ffb") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:14.643415 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:14.643372    2583 generic.go:356] "Generic (PLEG): container finished" podID="a41b66ba-345d-4b30-9987-043f67724e6e" containerID="a864bf9b6137eaf8f45812ad138d8c64279862fe4493eaf15294b5a7fbd6e0d4" exitCode=0
Sep 17 19:07:14.643868 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:14.643459    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j74f8" event={"ID":"a41b66ba-345d-4b30-9987-043f67724e6e","Type":"ContainerDied","Data":"a864bf9b6137eaf8f45812ad138d8c64279862fe4493eaf15294b5a7fbd6e0d4"}
Sep 17 19:07:15.072227 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:15.072161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:15.072227 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:15.072210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:15.072462 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:15.072310    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:07:15.072462 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:15.072382    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls podName:92e4146f-3a14-40d3-bbb5-83bbe2f8be67 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:19.072359953 +0000 UTC m=+42.900218168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls") pod "dns-default-66n6k" (UID: "92e4146f-3a14-40d3-bbb5-83bbe2f8be67") : secret "dns-default-metrics-tls" not found
Sep 17 19:07:15.072462 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:15.072389    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:07:15.072462 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:15.072403    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648597d777-lfb4m: secret "image-registry-tls" not found
Sep 17 19:07:15.072462 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:15.072464    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls podName:74441563-19c2-4fe5-8802-9db0426c73f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:19.072437273 +0000 UTC m=+42.900295506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls") pod "image-registry-648597d777-lfb4m" (UID: "74441563-19c2-4fe5-8802-9db0426c73f3") : secret "image-registry-tls" not found
Sep 17 19:07:15.560302 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:15.560181    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kdh4s_f6cc77d6-c367-470c-bfca-7399dc10d65b/dns-node-resolver/0.log"
Sep 17 19:07:15.578164 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:15.578127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:15.578340 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:15.578192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:15.578340 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:15.578320    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:15.578489 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:15.578378    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls podName:b65a92c1-7983-4dc0-afd3-eed290a55ffb nodeName:}" failed. No retries permitted until 2026-09-17 19:07:17.578360058 +0000 UTC m=+41.406218274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4c96" (UID: "b65a92c1-7983-4dc0-afd3-eed290a55ffb") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:15.578576 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:15.578533    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:07:15.578636 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:15.578586    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm podName:b65a92c1-7983-4dc0-afd3-eed290a55ffb nodeName:}" failed. No retries permitted until 2026-09-17 19:07:17.578571164 +0000 UTC m=+41.406429391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x4c96" (UID: "b65a92c1-7983-4dc0-afd3-eed290a55ffb") : configmap "kube-rbac-proxy" not found
Sep 17 19:07:15.653262 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:15.652657    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j74f8" event={"ID":"a41b66ba-345d-4b30-9987-043f67724e6e","Type":"ContainerStarted","Data":"ce300c805abb9900175e3dc7aa41306df2b4dd8a54fd3e51e10f8e7ceb447d2b"}
Sep 17 19:07:15.681535 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:15.681401    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j74f8" podStartSLOduration=10.198288069 podStartE2EDuration="37.681385981s" totalImagesPullingTime="27.483097912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:15.679982746 +0000 UTC m=+39.507840982" watchObservedRunningTime="2026-09-17 19:07:15.681385981 +0000 UTC m=+39.509244239"
Sep 17 19:07:16.434379 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:16.434341    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-99d9m"]
Sep 17 19:07:16.436864 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:16.436838    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-99d9m"
Sep 17 19:07:16.439382 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:16.439352    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-897g8\""
Sep 17 19:07:16.451094 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:16.451072    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-99d9m"]
Sep 17 19:07:16.588771 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:16.588729    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfvct\" (UniqueName: \"kubernetes.io/projected/29449e74-88a3-41dd-bc8f-042b26ad1055-kube-api-access-gfvct\") pod \"network-check-source-7cbc45fddc-99d9m\" (UID: \"29449e74-88a3-41dd-bc8f-042b26ad1055\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-99d9m"
Sep 17 19:07:16.689716 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:16.689634    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfvct\" (UniqueName: \"kubernetes.io/projected/29449e74-88a3-41dd-bc8f-042b26ad1055-kube-api-access-gfvct\") pod \"network-check-source-7cbc45fddc-99d9m\" (UID: \"29449e74-88a3-41dd-bc8f-042b26ad1055\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-99d9m"
Sep 17 19:07:16.699873 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:16.699839    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfvct\" (UniqueName: \"kubernetes.io/projected/29449e74-88a3-41dd-bc8f-042b26ad1055-kube-api-access-gfvct\") pod \"network-check-source-7cbc45fddc-99d9m\" (UID: \"29449e74-88a3-41dd-bc8f-042b26ad1055\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-99d9m"
Sep 17 19:07:16.748659 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:16.748629    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-99d9m"
Sep 17 19:07:16.758917 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:16.758896    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m68t4_68a1d862-0402-4b78-a9fc-d61d8b1816ac/node-ca/0.log"
Sep 17 19:07:17.596504 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:17.596461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:17.596683 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:17.596596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:17.596683 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:17.596648    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:17.596809 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:17.596693    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:07:17.596809 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:17.596731    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls podName:b65a92c1-7983-4dc0-afd3-eed290a55ffb nodeName:}" failed. No retries permitted until 2026-09-17 19:07:21.59670965 +0000 UTC m=+45.424567882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4c96" (UID: "b65a92c1-7983-4dc0-afd3-eed290a55ffb") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:17.596809 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:17.596752    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm podName:b65a92c1-7983-4dc0-afd3-eed290a55ffb nodeName:}" failed. No retries permitted until 2026-09-17 19:07:21.596741457 +0000 UTC m=+45.424599672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x4c96" (UID: "b65a92c1-7983-4dc0-afd3-eed290a55ffb") : configmap "kube-rbac-proxy" not found
Sep 17 19:07:18.583707 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:18.583672    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-99d9m"]
Sep 17 19:07:18.588271 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:18.588244    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29449e74_88a3_41dd_bc8f_042b26ad1055.slice/crio-9329a48c540bba3a7cba08e589eccf338e15b4471b145cd9ff376dff77f44bc2 WatchSource:0}: Error finding container 9329a48c540bba3a7cba08e589eccf338e15b4471b145cd9ff376dff77f44bc2: Status 404 returned error can't find the container with id 9329a48c540bba3a7cba08e589eccf338e15b4471b145cd9ff376dff77f44bc2
Sep 17 19:07:18.660924 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:18.660895    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-99d9m" event={"ID":"29449e74-88a3-41dd-bc8f-042b26ad1055","Type":"ContainerStarted","Data":"9329a48c540bba3a7cba08e589eccf338e15b4471b145cd9ff376dff77f44bc2"}
Sep 17 19:07:18.662332 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:18.662304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x5kmn" event={"ID":"6f451007-41b2-4c22-8441-36a2b6284c60","Type":"ContainerStarted","Data":"5fb378fd3847f8098b70f8e08ba8ab3acf2ab29f3bb6bdc4e16da77c03bccb59"}
Sep 17 19:07:18.663732 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:18.663711    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sv444" event={"ID":"f78a6d46-2733-4c1a-83a9-e566098efa57","Type":"ContainerStarted","Data":"9e8a546989e13a3f1a545b17742b95cbe2ff38c2be2646f3d8f2ae262f8e72d4"}
Sep 17 19:07:18.663838 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:18.663794    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:07:18.665068 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:18.665049    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vtv4x" event={"ID":"14e87624-a049-426c-b0ef-8d21d1100337","Type":"ContainerStarted","Data":"3b1d8ed6474773dc23fca107f9614fe52399407e66e544c89b6784581482544d"}
Sep 17 19:07:18.693951 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:18.693893    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x5kmn" podStartSLOduration=35.353881953 podStartE2EDuration="40.693873287s" totalImagesPullingTime="5.339991334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:18.693561167 +0000 UTC m=+42.521419402" watchObservedRunningTime="2026-09-17 19:07:18.693873287 +0000 UTC m=+42.521731523"
Sep 17 19:07:18.753776 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:18.753670    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sv444" podStartSLOduration=35.424457227 podStartE2EDuration="40.753653559s" totalImagesPullingTime="5.329196332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:18.745583639 +0000 UTC m=+42.573441873" watchObservedRunningTime="2026-09-17 19:07:18.753653559 +0000 UTC m=+42.581511793"
Sep 17 19:07:18.777524 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:18.777478    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vtv4x" podStartSLOduration=2.446120121 podStartE2EDuration="7.777462606s" totalImagesPullingTime="5.331342485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:18.776658705 +0000 UTC m=+42.604516940" watchObservedRunningTime="2026-09-17 19:07:18.777462606 +0000 UTC m=+42.605320832"
Sep 17 19:07:18.861104 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:18.861066    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-knwrz"]
Sep 17 19:07:18.882180 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:18.882145    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-knwrz"
Sep 17 19:07:18.885216 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:18.885187    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-knwrz"]
Sep 17 19:07:18.885592 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:18.885575    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 19:07:18.885684 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:18.885579    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 19:07:18.886707 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:18.886687    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4sg2j\""
Sep 17 19:07:19.009703 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:19.009626    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlwr7\" (UniqueName: \"kubernetes.io/projected/6b111e2f-6f34-4f14-9e41-119a5d6e112b-kube-api-access-vlwr7\") pod \"migrator-76794cbd6c-knwrz\" (UID: \"6b111e2f-6f34-4f14-9e41-119a5d6e112b\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-knwrz"
Sep 17 19:07:19.110926 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:19.110885    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:19.110926 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:19.110928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:19.111122 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:19.110965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlwr7\" (UniqueName: \"kubernetes.io/projected/6b111e2f-6f34-4f14-9e41-119a5d6e112b-kube-api-access-vlwr7\") pod \"migrator-76794cbd6c-knwrz\" (UID: \"6b111e2f-6f34-4f14-9e41-119a5d6e112b\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-knwrz"
Sep 17 19:07:19.111122 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:19.111052    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:07:19.111122 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:19.111069    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:07:19.111122 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:19.111090    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648597d777-lfb4m: secret "image-registry-tls" not found
Sep 17 19:07:19.111122 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:19.111115    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls podName:92e4146f-3a14-40d3-bbb5-83bbe2f8be67 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:27.11109811 +0000 UTC m=+50.938956325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls") pod "dns-default-66n6k" (UID: "92e4146f-3a14-40d3-bbb5-83bbe2f8be67") : secret "dns-default-metrics-tls" not found
Sep 17 19:07:19.111287 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:19.111138    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls podName:74441563-19c2-4fe5-8802-9db0426c73f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:27.111121274 +0000 UTC m=+50.938979510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls") pod "image-registry-648597d777-lfb4m" (UID: "74441563-19c2-4fe5-8802-9db0426c73f3") : secret "image-registry-tls" not found
Sep 17 19:07:19.122974 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:19.122947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlwr7\" (UniqueName: \"kubernetes.io/projected/6b111e2f-6f34-4f14-9e41-119a5d6e112b-kube-api-access-vlwr7\") pod \"migrator-76794cbd6c-knwrz\" (UID: \"6b111e2f-6f34-4f14-9e41-119a5d6e112b\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-knwrz"
Sep 17 19:07:19.192143 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:19.192113    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-knwrz"
Sep 17 19:07:19.497209 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:19.497179    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-knwrz"]
Sep 17 19:07:19.499858 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:19.499830    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b111e2f_6f34_4f14_9e41_119a5d6e112b.slice/crio-3f4c9a8212157a809470505dfb9d92f072cfe8260d19e89bf066a5593b9b1d19 WatchSource:0}: Error finding container 3f4c9a8212157a809470505dfb9d92f072cfe8260d19e89bf066a5593b9b1d19: Status 404 returned error can't find the container with id 3f4c9a8212157a809470505dfb9d92f072cfe8260d19e89bf066a5593b9b1d19
Sep 17 19:07:19.669495 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:19.669457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-99d9m" event={"ID":"29449e74-88a3-41dd-bc8f-042b26ad1055","Type":"ContainerStarted","Data":"f4567d4c5fe1d232be37f86f0a627a6b6ed258cb9182cc03becfa6bf7740024d"}
Sep 17 19:07:19.670619 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:19.670598    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-knwrz" event={"ID":"6b111e2f-6f34-4f14-9e41-119a5d6e112b","Type":"ContainerStarted","Data":"3f4c9a8212157a809470505dfb9d92f072cfe8260d19e89bf066a5593b9b1d19"}
Sep 17 19:07:19.689455 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:19.689398    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-99d9m" podStartSLOduration=3.6893810929999997 podStartE2EDuration="3.689381093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:19.688964761 +0000 UTC m=+43.516822995" watchObservedRunningTime="2026-09-17 19:07:19.689381093 +0000 UTC m=+43.517239330"
Sep 17 19:07:20.675888 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:20.675853    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-knwrz" event={"ID":"6b111e2f-6f34-4f14-9e41-119a5d6e112b","Type":"ContainerStarted","Data":"d7a10cc48777eb6b125d27268d2803d9a953c4df53448f5366008300c822fa8e"}
Sep 17 19:07:20.676264 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:20.675907    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-knwrz" event={"ID":"6b111e2f-6f34-4f14-9e41-119a5d6e112b","Type":"ContainerStarted","Data":"c4ddff067cdc3a42006a5d45040544fb56def7581045530fe264d1b138f24ec8"}
Sep 17 19:07:20.714033 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:20.713971    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-knwrz" podStartSLOduration=1.733802522 podStartE2EDuration="2.713948322s" totalImagesPullingTime="980.1458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:20.713143916 +0000 UTC m=+44.541002154" watchObservedRunningTime="2026-09-17 19:07:20.713948322 +0000 UTC m=+44.541806557"
Sep 17 19:07:21.631770 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:21.631726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:21.631911 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:21.631828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:21.631911 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:21.631898    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:21.632003 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:21.631927    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:07:21.632003 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:21.631970    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm podName:b65a92c1-7983-4dc0-afd3-eed290a55ffb nodeName:}" failed. No retries permitted until 2026-09-17 19:07:29.631953926 +0000 UTC m=+53.459812158 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x4c96" (UID: "b65a92c1-7983-4dc0-afd3-eed290a55ffb") : configmap "kube-rbac-proxy" not found
Sep 17 19:07:21.632003 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:21.631986    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls podName:b65a92c1-7983-4dc0-afd3-eed290a55ffb nodeName:}" failed. No retries permitted until 2026-09-17 19:07:29.631977308 +0000 UTC m=+53.459835521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x4c96" (UID: "b65a92c1-7983-4dc0-afd3-eed290a55ffb") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:27.169456 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:27.169414    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:27.169983 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:27.169468    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:27.169983 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:27.169581    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 19:07:27.169983 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:27.169586    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:07:27.169983 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:27.169663    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls podName:92e4146f-3a14-40d3-bbb5-83bbe2f8be67 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:43.169644178 +0000 UTC m=+66.997502394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls") pod "dns-default-66n6k" (UID: "92e4146f-3a14-40d3-bbb5-83bbe2f8be67") : secret "dns-default-metrics-tls" not found
Sep 17 19:07:27.169983 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:27.169596    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648597d777-lfb4m: secret "image-registry-tls" not found
Sep 17 19:07:27.169983 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:27.169728    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls podName:74441563-19c2-4fe5-8802-9db0426c73f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:43.169715984 +0000 UTC m=+66.997574197 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls") pod "image-registry-648597d777-lfb4m" (UID: "74441563-19c2-4fe5-8802-9db0426c73f3") : secret "image-registry-tls" not found
Sep 17 19:07:29.685988 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:29.685947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:29.686469 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:29.685998    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:29.686569 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:29.686550    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b65a92c1-7983-4dc0-afd3-eed290a55ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:29.688292 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:29.688271    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b65a92c1-7983-4dc0-afd3-eed290a55ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4c96\" (UID: \"b65a92c1-7983-4dc0-afd3-eed290a55ffb\") " pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:29.757233 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:29.757212    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x4c96"
Sep 17 19:07:30.070549 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:30.070465    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x4c96"]
Sep 17 19:07:30.072985 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:30.072957    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65a92c1_7983_4dc0_afd3_eed290a55ffb.slice/crio-52cfa9b758296e0a3bb10bc810ebbb294664741a4373036f43574357459f14c8 WatchSource:0}: Error finding container 52cfa9b758296e0a3bb10bc810ebbb294664741a4373036f43574357459f14c8: Status 404 returned error can't find the container with id 52cfa9b758296e0a3bb10bc810ebbb294664741a4373036f43574357459f14c8
Sep 17 19:07:30.711724 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:30.711682    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4c96" event={"ID":"b65a92c1-7983-4dc0-afd3-eed290a55ffb","Type":"ContainerStarted","Data":"bc5bfd9134509ac3f0514cd973bd3898e6abfa85adf97b23596414436978b5f5"}
Sep 17 19:07:30.712068 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:30.711734    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4c96" event={"ID":"b65a92c1-7983-4dc0-afd3-eed290a55ffb","Type":"ContainerStarted","Data":"52cfa9b758296e0a3bb10bc810ebbb294664741a4373036f43574357459f14c8"}
Sep 17 19:07:31.718021 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:31.717980    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4c96" event={"ID":"b65a92c1-7983-4dc0-afd3-eed290a55ffb","Type":"ContainerStarted","Data":"110b1949f3b262890f7f61d9fe9bb4b472a8fdfed3536e0602e11c7d2aecfdcf"}
Sep 17 19:07:33.731089 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:33.731052    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4c96" event={"ID":"b65a92c1-7983-4dc0-afd3-eed290a55ffb","Type":"ContainerStarted","Data":"7e4ed17945e0f2c6fe37572e16317d71112fae53839544a0109271bea9a76ce3"}
Sep 17 19:07:35.618703 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:35.618675    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9lk9z"
Sep 17 19:07:35.652782 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:35.652730    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x4c96" podStartSLOduration=20.028316983 podStartE2EDuration="22.652712544s" totalImagesPullingTime="2.624395561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:07:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:33.752403825 +0000 UTC m=+57.580262060" watchObservedRunningTime="2026-09-17 19:07:35.652712544 +0000 UTC m=+59.480570778"
Sep 17 19:07:43.176202 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:43.176145    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:43.176202 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:43.176203    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:43.178552 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:43.178530    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/92e4146f-3a14-40d3-bbb5-83bbe2f8be67-metrics-tls\") pod \"dns-default-66n6k\" (UID: \"92e4146f-3a14-40d3-bbb5-83bbe2f8be67\") " pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:43.178681 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:43.178660    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls\") pod \"image-registry-648597d777-lfb4m\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") " pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:43.460021 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:43.459945    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-msk9c\""
Sep 17 19:07:43.467351 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:43.467330    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:43.469913 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:43.469893    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dpmzt\""
Sep 17 19:07:43.478139 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:43.478114    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:43.740850 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:43.740778    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-648597d777-lfb4m"]
Sep 17 19:07:43.743507 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:43.743480    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74441563_19c2_4fe5_8802_9db0426c73f3.slice/crio-3b3890d758e30e5b6ae85e48fb34fcfbcd067c433a1d9c9555d6f814c6f3ab56 WatchSource:0}: Error finding container 3b3890d758e30e5b6ae85e48fb34fcfbcd067c433a1d9c9555d6f814c6f3ab56: Status 404 returned error can't find the container with id 3b3890d758e30e5b6ae85e48fb34fcfbcd067c433a1d9c9555d6f814c6f3ab56
Sep 17 19:07:43.760168 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:43.760144    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648597d777-lfb4m" event={"ID":"74441563-19c2-4fe5-8802-9db0426c73f3","Type":"ContainerStarted","Data":"3b3890d758e30e5b6ae85e48fb34fcfbcd067c433a1d9c9555d6f814c6f3ab56"}
Sep 17 19:07:43.762258 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:43.762196    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-66n6k"]
Sep 17 19:07:43.779648 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:43.779617    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e4146f_3a14_40d3_bbb5_83bbe2f8be67.slice/crio-b46799482bfb1d98f84dfcedaab832cc3eaf9045ba6ab55505315d4e972c5558 WatchSource:0}: Error finding container b46799482bfb1d98f84dfcedaab832cc3eaf9045ba6ab55505315d4e972c5558: Status 404 returned error can't find the container with id b46799482bfb1d98f84dfcedaab832cc3eaf9045ba6ab55505315d4e972c5558
Sep 17 19:07:44.183427 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:44.183387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs\") pod \"network-metrics-daemon-r2lm8\" (UID: \"d7c98728-9f29-4061-9131-95535a68e5c0\") " pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:07:44.186552 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:44.186528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7c98728-9f29-4061-9131-95535a68e5c0-metrics-certs\") pod \"network-metrics-daemon-r2lm8\" (UID: \"d7c98728-9f29-4061-9131-95535a68e5c0\") " pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:07:44.469025 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:44.468942    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-47zq2\""
Sep 17 19:07:44.477235 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:44.477208    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2lm8"
Sep 17 19:07:44.751654 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:44.751563    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r2lm8"]
Sep 17 19:07:44.755633 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:44.755600    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c98728_9f29_4061_9131_95535a68e5c0.slice/crio-f33d751aa982aa4f50c65ed440cdc3c842e7277a7025cc8c49ac6e43e4ac5e9e WatchSource:0}: Error finding container f33d751aa982aa4f50c65ed440cdc3c842e7277a7025cc8c49ac6e43e4ac5e9e: Status 404 returned error can't find the container with id f33d751aa982aa4f50c65ed440cdc3c842e7277a7025cc8c49ac6e43e4ac5e9e
Sep 17 19:07:44.765573 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:44.765547    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648597d777-lfb4m" event={"ID":"74441563-19c2-4fe5-8802-9db0426c73f3","Type":"ContainerStarted","Data":"a243262146953cc04536e5fbb4e2619933ecd49f31f5a28712cc9983f55a3a0f"}
Sep 17 19:07:44.765689 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:44.765647    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:07:44.766781 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:44.766758    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r2lm8" event={"ID":"d7c98728-9f29-4061-9131-95535a68e5c0","Type":"ContainerStarted","Data":"f33d751aa982aa4f50c65ed440cdc3c842e7277a7025cc8c49ac6e43e4ac5e9e"}
Sep 17 19:07:44.767733 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:44.767711    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-66n6k" event={"ID":"92e4146f-3a14-40d3-bbb5-83bbe2f8be67","Type":"ContainerStarted","Data":"b46799482bfb1d98f84dfcedaab832cc3eaf9045ba6ab55505315d4e972c5558"}
Sep 17 19:07:44.793009 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:44.792955    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-648597d777-lfb4m" podStartSLOduration=34.792939537 podStartE2EDuration="34.792939537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:44.792393326 +0000 UTC m=+68.620251562" watchObservedRunningTime="2026-09-17 19:07:44.792939537 +0000 UTC m=+68.620797771"
Sep 17 19:07:45.774269 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:45.774238    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-66n6k" event={"ID":"92e4146f-3a14-40d3-bbb5-83bbe2f8be67","Type":"ContainerStarted","Data":"2e2f973387a93f0815272ccb5494104cd7308627dfee69a1e08228f56ea6f6c7"}
Sep 17 19:07:45.774712 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:45.774328    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-66n6k" event={"ID":"92e4146f-3a14-40d3-bbb5-83bbe2f8be67","Type":"ContainerStarted","Data":"b139c9607053f1471ee922c4547fda5949ac7241447eee80c0ea9e3c583c25de"}
Sep 17 19:07:45.774712 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:45.774566    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:45.796160 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:45.796108    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-66n6k" podStartSLOduration=33.334997758 podStartE2EDuration="34.796093047s" totalImagesPullingTime="1.461095289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:45.795331935 +0000 UTC m=+69.623190172" watchObservedRunningTime="2026-09-17 19:07:45.796093047 +0000 UTC m=+69.623951282"
Sep 17 19:07:46.295859 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.295827    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"]
Sep 17 19:07:46.297722 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.297706    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"
Sep 17 19:07:46.300755 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.300687    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 19:07:46.300894 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.300786    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 19:07:46.301738 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.301721    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 19:07:46.301738 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.301731    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 19:07:46.318220 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.318193    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"]
Sep 17 19:07:46.365876 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.365843    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-648597d777-lfb4m"]
Sep 17 19:07:46.371857 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.371829    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w8d54"]
Sep 17 19:07:46.373990 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.373974    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-lcwsh"]
Sep 17 19:07:46.374103 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.374086    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w8d54"
Sep 17 19:07:46.375934 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.375906    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-lcwsh"
Sep 17 19:07:46.376933 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.376915    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 19:07:46.377062 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.377038    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5tt7g\""
Sep 17 19:07:46.378421 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.378404    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 19:07:46.378565 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.378550    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 19:07:46.378689 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.378675    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-8dm87\""
Sep 17 19:07:46.386614 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.386592    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w8d54"]
Sep 17 19:07:46.388865 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.388844    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-lcwsh"]
Sep 17 19:07:46.398379 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.398145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4p9g\" (UniqueName: \"kubernetes.io/projected/4c02e5ba-0de8-4429-9ce9-502dcb0cf90b-kube-api-access-k4p9g\") pod \"klusterlet-addon-workmgr-6cdddb89dd-bhhm5\" (UID: \"4c02e5ba-0de8-4429-9ce9-502dcb0cf90b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"
Sep 17 19:07:46.398379 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.398195    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c02e5ba-0de8-4429-9ce9-502dcb0cf90b-tmp\") pod \"klusterlet-addon-workmgr-6cdddb89dd-bhhm5\" (UID: \"4c02e5ba-0de8-4429-9ce9-502dcb0cf90b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"
Sep 17 19:07:46.398379 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.398239    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4c02e5ba-0de8-4429-9ce9-502dcb0cf90b-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cdddb89dd-bhhm5\" (UID: \"4c02e5ba-0de8-4429-9ce9-502dcb0cf90b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"
Sep 17 19:07:46.398379 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.398269    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e6a0e9a3-4309-42a4-bc21-c5131f231ab6-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-w8d54\" (UID: \"e6a0e9a3-4309-42a4-bc21-c5131f231ab6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w8d54"
Sep 17 19:07:46.398379 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.398293    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26rr\" (UniqueName: \"kubernetes.io/projected/a4457e63-37ef-4ef9-8cc0-47028302cb94-kube-api-access-g26rr\") pod \"downloads-6594f4c75-lcwsh\" (UID: \"a4457e63-37ef-4ef9-8cc0-47028302cb94\") " pod="openshift-console/downloads-6594f4c75-lcwsh"
Sep 17 19:07:46.483867 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.483835    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c588ccb4c-d45nx"]
Sep 17 19:07:46.485939 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.485918    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.498760 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.498726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c02e5ba-0de8-4429-9ce9-502dcb0cf90b-tmp\") pod \"klusterlet-addon-workmgr-6cdddb89dd-bhhm5\" (UID: \"4c02e5ba-0de8-4429-9ce9-502dcb0cf90b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"
Sep 17 19:07:46.498899 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.498784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4c02e5ba-0de8-4429-9ce9-502dcb0cf90b-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cdddb89dd-bhhm5\" (UID: \"4c02e5ba-0de8-4429-9ce9-502dcb0cf90b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"
Sep 17 19:07:46.498899 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.498825    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e6a0e9a3-4309-42a4-bc21-c5131f231ab6-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-w8d54\" (UID: \"e6a0e9a3-4309-42a4-bc21-c5131f231ab6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w8d54"
Sep 17 19:07:46.498899 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.498853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26rr\" (UniqueName: \"kubernetes.io/projected/a4457e63-37ef-4ef9-8cc0-47028302cb94-kube-api-access-g26rr\") pod \"downloads-6594f4c75-lcwsh\" (UID: \"a4457e63-37ef-4ef9-8cc0-47028302cb94\") " pod="openshift-console/downloads-6594f4c75-lcwsh"
Sep 17 19:07:46.498899 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.498886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4p9g\" (UniqueName: \"kubernetes.io/projected/4c02e5ba-0de8-4429-9ce9-502dcb0cf90b-kube-api-access-k4p9g\") pod \"klusterlet-addon-workmgr-6cdddb89dd-bhhm5\" (UID: \"4c02e5ba-0de8-4429-9ce9-502dcb0cf90b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"
Sep 17 19:07:46.499110 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.499050    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c02e5ba-0de8-4429-9ce9-502dcb0cf90b-tmp\") pod \"klusterlet-addon-workmgr-6cdddb89dd-bhhm5\" (UID: \"4c02e5ba-0de8-4429-9ce9-502dcb0cf90b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"
Sep 17 19:07:46.501105 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.501084    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e6a0e9a3-4309-42a4-bc21-c5131f231ab6-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-w8d54\" (UID: \"e6a0e9a3-4309-42a4-bc21-c5131f231ab6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w8d54"
Sep 17 19:07:46.501105 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.501097    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4c02e5ba-0de8-4429-9ce9-502dcb0cf90b-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cdddb89dd-bhhm5\" (UID: \"4c02e5ba-0de8-4429-9ce9-502dcb0cf90b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"
Sep 17 19:07:46.504722 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.504697    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c588ccb4c-d45nx"]
Sep 17 19:07:46.514919 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.514889    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26rr\" (UniqueName: \"kubernetes.io/projected/a4457e63-37ef-4ef9-8cc0-47028302cb94-kube-api-access-g26rr\") pod \"downloads-6594f4c75-lcwsh\" (UID: \"a4457e63-37ef-4ef9-8cc0-47028302cb94\") " pod="openshift-console/downloads-6594f4c75-lcwsh"
Sep 17 19:07:46.515040 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.514994    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4p9g\" (UniqueName: \"kubernetes.io/projected/4c02e5ba-0de8-4429-9ce9-502dcb0cf90b-kube-api-access-k4p9g\") pod \"klusterlet-addon-workmgr-6cdddb89dd-bhhm5\" (UID: \"4c02e5ba-0de8-4429-9ce9-502dcb0cf90b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"
Sep 17 19:07:46.600027 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.599939    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fad2458d-4ef5-4d9b-99c4-e323be51944e-registry-tls\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.600027 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.599980    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fad2458d-4ef5-4d9b-99c4-e323be51944e-bound-sa-token\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.600027 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.600013    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fad2458d-4ef5-4d9b-99c4-e323be51944e-registry-certificates\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.600241 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.600042    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fad2458d-4ef5-4d9b-99c4-e323be51944e-ca-trust-extracted\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.600241 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.600062    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fad2458d-4ef5-4d9b-99c4-e323be51944e-image-registry-private-configuration\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.600241 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.600154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km5px\" (UniqueName: \"kubernetes.io/projected/fad2458d-4ef5-4d9b-99c4-e323be51944e-kube-api-access-km5px\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.600241 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.600185    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fad2458d-4ef5-4d9b-99c4-e323be51944e-trusted-ca\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.600241 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.600215    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fad2458d-4ef5-4d9b-99c4-e323be51944e-installation-pull-secrets\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.607978 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.607943    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"
Sep 17 19:07:46.683766 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.683732    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w8d54"
Sep 17 19:07:46.689505 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.689478    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-lcwsh"
Sep 17 19:07:46.700681 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.700649    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fad2458d-4ef5-4d9b-99c4-e323be51944e-ca-trust-extracted\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.700806 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.700701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fad2458d-4ef5-4d9b-99c4-e323be51944e-image-registry-private-configuration\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.700806 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.700739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km5px\" (UniqueName: \"kubernetes.io/projected/fad2458d-4ef5-4d9b-99c4-e323be51944e-kube-api-access-km5px\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.700806 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.700768    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fad2458d-4ef5-4d9b-99c4-e323be51944e-trusted-ca\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.700806 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.700795    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fad2458d-4ef5-4d9b-99c4-e323be51944e-installation-pull-secrets\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.701039 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.700824    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fad2458d-4ef5-4d9b-99c4-e323be51944e-registry-tls\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.701039 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.700863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fad2458d-4ef5-4d9b-99c4-e323be51944e-bound-sa-token\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.701039 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.700898    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fad2458d-4ef5-4d9b-99c4-e323be51944e-registry-certificates\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.701205 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.701039    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fad2458d-4ef5-4d9b-99c4-e323be51944e-ca-trust-extracted\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.701717 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.701692    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fad2458d-4ef5-4d9b-99c4-e323be51944e-registry-certificates\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.702139 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.702106    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fad2458d-4ef5-4d9b-99c4-e323be51944e-trusted-ca\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.703714 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.703690    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fad2458d-4ef5-4d9b-99c4-e323be51944e-image-registry-private-configuration\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.704111 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.704088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fad2458d-4ef5-4d9b-99c4-e323be51944e-installation-pull-secrets\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.704174 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.704133    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fad2458d-4ef5-4d9b-99c4-e323be51944e-registry-tls\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.730099 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.729802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km5px\" (UniqueName: \"kubernetes.io/projected/fad2458d-4ef5-4d9b-99c4-e323be51944e-kube-api-access-km5px\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.730974 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.730645    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fad2458d-4ef5-4d9b-99c4-e323be51944e-bound-sa-token\") pod \"image-registry-5c588ccb4c-d45nx\" (UID: \"fad2458d-4ef5-4d9b-99c4-e323be51944e\") " pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.779485 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.779435    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r2lm8" event={"ID":"d7c98728-9f29-4061-9131-95535a68e5c0","Type":"ContainerStarted","Data":"c1d571c178a3b2e7b80d0c96ba9316ad9fa115881ab54d6f87c72582236a6f3c"}
Sep 17 19:07:46.779888 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.779513    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r2lm8" event={"ID":"d7c98728-9f29-4061-9131-95535a68e5c0","Type":"ContainerStarted","Data":"849379078e3edaab0c422f6ff0a1dc28981415cae7f6c8decdef1dfbd052949a"}
Sep 17 19:07:46.794224 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.794199    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:46.835465 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.835390    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r2lm8" podStartSLOduration=67.840223247 podStartE2EDuration="1m8.835366199s" totalImagesPullingTime="995.142952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:46.834530679 +0000 UTC m=+70.662388913" watchObservedRunningTime="2026-09-17 19:07:46.835366199 +0000 UTC m=+70.663224435"
Sep 17 19:07:46.880800 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.880691    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"]
Sep 17 19:07:46.885630 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:46.885602    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c02e5ba_0de8_4429_9ce9_502dcb0cf90b.slice/crio-94d7dc277651789718a7edcb59b653fef23cf9c9018808b6c4610969f26e715b WatchSource:0}: Error finding container 94d7dc277651789718a7edcb59b653fef23cf9c9018808b6c4610969f26e715b: Status 404 returned error can't find the container with id 94d7dc277651789718a7edcb59b653fef23cf9c9018808b6c4610969f26e715b
Sep 17 19:07:46.969310 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.969270    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w8d54"]
Sep 17 19:07:46.972091 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:46.972067    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a0e9a3_4309_42a4_bc21_c5131f231ab6.slice/crio-a2cce7bf07772b95508109b6a1d883cf3c38129694790df8f0c7769f1c3874a2 WatchSource:0}: Error finding container a2cce7bf07772b95508109b6a1d883cf3c38129694790df8f0c7769f1c3874a2: Status 404 returned error can't find the container with id a2cce7bf07772b95508109b6a1d883cf3c38129694790df8f0c7769f1c3874a2
Sep 17 19:07:46.985493 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:46.985466    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-lcwsh"]
Sep 17 19:07:46.988758 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:46.988733    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4457e63_37ef_4ef9_8cc0_47028302cb94.slice/crio-d765a9d62e898afdc85a80f81f6188b2dcbdd09c5f5ca9aa7ef1844f6f041aac WatchSource:0}: Error finding container d765a9d62e898afdc85a80f81f6188b2dcbdd09c5f5ca9aa7ef1844f6f041aac: Status 404 returned error can't find the container with id d765a9d62e898afdc85a80f81f6188b2dcbdd09c5f5ca9aa7ef1844f6f041aac
Sep 17 19:07:47.086651 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.086615    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c588ccb4c-d45nx"]
Sep 17 19:07:47.089725 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:47.089702    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad2458d_4ef5_4d9b_99c4_e323be51944e.slice/crio-e65d4a762af42d95cde3faf666b081c77137ba8b512931390bfd50ef3270748f WatchSource:0}: Error finding container e65d4a762af42d95cde3faf666b081c77137ba8b512931390bfd50ef3270748f: Status 404 returned error can't find the container with id e65d4a762af42d95cde3faf666b081c77137ba8b512931390bfd50ef3270748f
Sep 17 19:07:47.108030 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.108008    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5468cc68cd-kq4g5"]
Sep 17 19:07:47.110777 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.110762    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.113720 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.113701    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 19:07:47.114043 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.114025    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 19:07:47.114129 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.114060    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 19:07:47.115004 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.114989    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qv7gn\""
Sep 17 19:07:47.115004 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.114995    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 19:07:47.117574 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.117559    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 19:07:47.125185 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.125146    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5468cc68cd-kq4g5"]
Sep 17 19:07:47.205743 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.205706    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-oauth-serving-cert\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.205743 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.205743    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-service-ca\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.205950 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.205776    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe304912-bbb0-4c00-8f44-885e054175c6-console-oauth-config\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.205950 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.205832    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-console-config\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.205950 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.205861    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe304912-bbb0-4c00-8f44-885e054175c6-console-serving-cert\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.205950 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.205881    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6brc\" (UniqueName: \"kubernetes.io/projected/fe304912-bbb0-4c00-8f44-885e054175c6-kube-api-access-s6brc\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.307069 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.307025    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe304912-bbb0-4c00-8f44-885e054175c6-console-serving-cert\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.307227 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.307076    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6brc\" (UniqueName: \"kubernetes.io/projected/fe304912-bbb0-4c00-8f44-885e054175c6-kube-api-access-s6brc\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.307227 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.307109    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-oauth-serving-cert\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.307227 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.307141    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-service-ca\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.307227 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.307197    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe304912-bbb0-4c00-8f44-885e054175c6-console-oauth-config\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.307409 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.307245    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-console-config\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.307855 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.307828    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-oauth-serving-cert\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.307985 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.307955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-service-ca\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.308082 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.308005    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-console-config\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.309605 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.309578    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe304912-bbb0-4c00-8f44-885e054175c6-console-oauth-config\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.309605 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.309595    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe304912-bbb0-4c00-8f44-885e054175c6-console-serving-cert\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.315741 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.315719    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6brc\" (UniqueName: \"kubernetes.io/projected/fe304912-bbb0-4c00-8f44-885e054175c6-kube-api-access-s6brc\") pod \"console-5468cc68cd-kq4g5\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") " pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.429851 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.429638    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:47.720654 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.720574    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5468cc68cd-kq4g5"]
Sep 17 19:07:47.725529 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:47.725480    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe304912_bbb0_4c00_8f44_885e054175c6.slice/crio-53ead9535703d97197db48a1b3f96237d4f4e9aee82482562c880528d7912bd0 WatchSource:0}: Error finding container 53ead9535703d97197db48a1b3f96237d4f4e9aee82482562c880528d7912bd0: Status 404 returned error can't find the container with id 53ead9535703d97197db48a1b3f96237d4f4e9aee82482562c880528d7912bd0
Sep 17 19:07:47.788660 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.787785    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx" event={"ID":"fad2458d-4ef5-4d9b-99c4-e323be51944e","Type":"ContainerStarted","Data":"73231116f43b21980708fa4cf9104b947a66e0969e2a181bfa05f52b9ab1a819"}
Sep 17 19:07:47.788660 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.787848    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx" event={"ID":"fad2458d-4ef5-4d9b-99c4-e323be51944e","Type":"ContainerStarted","Data":"e65d4a762af42d95cde3faf666b081c77137ba8b512931390bfd50ef3270748f"}
Sep 17 19:07:47.788660 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.788597    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:07:47.791086 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.791002    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w8d54" event={"ID":"e6a0e9a3-4309-42a4-bc21-c5131f231ab6","Type":"ContainerStarted","Data":"a2cce7bf07772b95508109b6a1d883cf3c38129694790df8f0c7769f1c3874a2"}
Sep 17 19:07:47.793330 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.793271    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5468cc68cd-kq4g5" event={"ID":"fe304912-bbb0-4c00-8f44-885e054175c6","Type":"ContainerStarted","Data":"53ead9535703d97197db48a1b3f96237d4f4e9aee82482562c880528d7912bd0"}
Sep 17 19:07:47.795471 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.795404    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5" event={"ID":"4c02e5ba-0de8-4429-9ce9-502dcb0cf90b","Type":"ContainerStarted","Data":"94d7dc277651789718a7edcb59b653fef23cf9c9018808b6c4610969f26e715b"}
Sep 17 19:07:47.798268 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.798219    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-lcwsh" event={"ID":"a4457e63-37ef-4ef9-8cc0-47028302cb94","Type":"ContainerStarted","Data":"d765a9d62e898afdc85a80f81f6188b2dcbdd09c5f5ca9aa7ef1844f6f041aac"}
Sep 17 19:07:47.815188 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:47.815136    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx" podStartSLOduration=1.815121207 podStartE2EDuration="1.815121207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:47.814254692 +0000 UTC m=+71.642112946" watchObservedRunningTime="2026-09-17 19:07:47.815121207 +0000 UTC m=+71.642979442"
Sep 17 19:07:49.202161 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.202017    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54bdd75bb4-fd24d"]
Sep 17 19:07:49.207159 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.207130    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.218696 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.218519    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 19:07:49.219325 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.219095    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54bdd75bb4-fd24d"]
Sep 17 19:07:49.323855 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.323487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c61d97a-58be-4e9f-8893-025eee26298e-console-oauth-config\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.323855 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.323553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnd9t\" (UniqueName: \"kubernetes.io/projected/5c61d97a-58be-4e9f-8893-025eee26298e-kube-api-access-nnd9t\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.323855 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.323606    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-service-ca\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.323855 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.323633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-console-config\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.323855 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.323661    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-trusted-ca-bundle\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.323855 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.323683    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-oauth-serving-cert\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.323855 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.323712    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c61d97a-58be-4e9f-8893-025eee26298e-console-serving-cert\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.424553 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.424513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c61d97a-58be-4e9f-8893-025eee26298e-console-serving-cert\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.424732 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.424603    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c61d97a-58be-4e9f-8893-025eee26298e-console-oauth-config\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.424732 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.424652    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnd9t\" (UniqueName: \"kubernetes.io/projected/5c61d97a-58be-4e9f-8893-025eee26298e-kube-api-access-nnd9t\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.424732 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.424701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-service-ca\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.424732 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.424728    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-console-config\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.424952 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.424756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-trusted-ca-bundle\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.424952 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.424778    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-oauth-serving-cert\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.425542 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.425514    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-oauth-serving-cert\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.425672 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.425642    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-service-ca\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.426098 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.426076    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-console-config\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.426807 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.426780    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-trusted-ca-bundle\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.434706 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.433808    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c61d97a-58be-4e9f-8893-025eee26298e-console-serving-cert\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.434706 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.434648    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c61d97a-58be-4e9f-8893-025eee26298e-console-oauth-config\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.436805 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.436782    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnd9t\" (UniqueName: \"kubernetes.io/projected/5c61d97a-58be-4e9f-8893-025eee26298e-kube-api-access-nnd9t\") pod \"console-54bdd75bb4-fd24d\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") " pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.523331 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.523244    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:49.676317 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.676292    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sv444"
Sep 17 19:07:49.825488 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.825354    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w8d54" event={"ID":"e6a0e9a3-4309-42a4-bc21-c5131f231ab6","Type":"ContainerStarted","Data":"6ad1ce9705fa1bb083e2b2d24d02505ccd9ea44afefa121b322331b8c9a2205d"}
Sep 17 19:07:49.825675 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.825566    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w8d54"
Sep 17 19:07:49.832695 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.832669    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w8d54"
Sep 17 19:07:49.877017 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.876941    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-w8d54" podStartSLOduration=1.266821334 podStartE2EDuration="3.876918434s" totalImagesPullingTime="2.6100971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:49.85332925 +0000 UTC m=+73.681187507" watchObservedRunningTime="2026-09-17 19:07:49.876918434 +0000 UTC m=+73.704776669"
Sep 17 19:07:49.877273 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:49.877117    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54bdd75bb4-fd24d"]
Sep 17 19:07:51.581247 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:51.581211    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c61d97a_58be_4e9f_8893_025eee26298e.slice/crio-7088534460bddf92713dde03dbefdb929b860bd556bebd056dc5a46e0e1974a3 WatchSource:0}: Error finding container 7088534460bddf92713dde03dbefdb929b860bd556bebd056dc5a46e0e1974a3: Status 404 returned error can't find the container with id 7088534460bddf92713dde03dbefdb929b860bd556bebd056dc5a46e0e1974a3
Sep 17 19:07:51.836272 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:51.836183    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54bdd75bb4-fd24d" event={"ID":"5c61d97a-58be-4e9f-8893-025eee26298e","Type":"ContainerStarted","Data":"7088534460bddf92713dde03dbefdb929b860bd556bebd056dc5a46e0e1974a3"}
Sep 17 19:07:54.854714 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:54.854473    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54bdd75bb4-fd24d" event={"ID":"5c61d97a-58be-4e9f-8893-025eee26298e","Type":"ContainerStarted","Data":"6cb92e997305cfd6a081d44478890aa1a5fcaa585298566cf50414fbb3c93403"}
Sep 17 19:07:54.861307 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:54.861275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5" event={"ID":"4c02e5ba-0de8-4429-9ce9-502dcb0cf90b","Type":"ContainerStarted","Data":"e7fe2f6d680c76512c3291dc0911ad2d81b497f799d7916f6b898ef1f26b662b"}
Sep 17 19:07:54.861822 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:54.861798    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"
Sep 17 19:07:54.863410 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:54.863360    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5"
Sep 17 19:07:54.877771 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:54.877616    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54bdd75bb4-fd24d" podStartSLOduration=2.706669927 podStartE2EDuration="5.877594142s" totalImagesPullingTime="3.170924215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:54.876503025 +0000 UTC m=+78.704361261" watchObservedRunningTime="2026-09-17 19:07:54.877594142 +0000 UTC m=+78.705452374"
Sep 17 19:07:54.897961 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:54.897891    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cdddb89dd-bhhm5" podStartSLOduration=1.075974113 podStartE2EDuration="8.897868574s" totalImagesPullingTime="7.821894461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:54.896299386 +0000 UTC m=+78.724157623" watchObservedRunningTime="2026-09-17 19:07:54.897868574 +0000 UTC m=+78.725726870"
Sep 17 19:07:54.927127 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:54.925938    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5468cc68cd-kq4g5" podStartSLOduration=0.879263705 podStartE2EDuration="7.925916403s" totalImagesPullingTime="7.046652698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:54.92302654 +0000 UTC m=+78.750884775" watchObservedRunningTime="2026-09-17 19:07:54.925916403 +0000 UTC m=+78.753774639"
Sep 17 19:07:55.320343 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.320313    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"]
Sep 17 19:07:55.323538 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.323515    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"
Sep 17 19:07:55.326667 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.326456    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 19:07:55.326667 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.326569    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 19:07:55.327519 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.327495    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 19:07:55.327648 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.327630    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 19:07:55.327730 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.327716    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-q26b6\""
Sep 17 19:07:55.328956 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.328937    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 19:07:55.349908 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.349874    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"]
Sep 17 19:07:55.356561 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.356530    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5fx8b"]
Sep 17 19:07:55.359210 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.359183    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.363862 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.363651    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2zq9r\""
Sep 17 19:07:55.363862 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.363823    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 19:07:55.365420 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.365222    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 19:07:55.365420 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.365384    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 19:07:55.382150 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.382108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7nkx\" (UniqueName: \"kubernetes.io/projected/3f0e9b4b-68cb-4902-832b-5907880fc427-kube-api-access-m7nkx\") pod \"openshift-state-metrics-86c5bc79d6-krhhj\" (UID: \"3f0e9b4b-68cb-4902-832b-5907880fc427\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"
Sep 17 19:07:55.382303 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.382166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f0e9b4b-68cb-4902-832b-5907880fc427-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-krhhj\" (UID: \"3f0e9b4b-68cb-4902-832b-5907880fc427\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"
Sep 17 19:07:55.382303 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.382268    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f0e9b4b-68cb-4902-832b-5907880fc427-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-krhhj\" (UID: \"3f0e9b4b-68cb-4902-832b-5907880fc427\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"
Sep 17 19:07:55.382393 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.382376    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f0e9b4b-68cb-4902-832b-5907880fc427-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-krhhj\" (UID: \"3f0e9b4b-68cb-4902-832b-5907880fc427\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"
Sep 17 19:07:55.483553 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.483503    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.483553 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.483558    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-694lp\" (UniqueName: \"kubernetes.io/projected/69e4e049-b862-482f-a97a-6755cd479291-kube-api-access-694lp\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.483814 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.483645    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-accelerators-collector-config\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.483814 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.483688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f0e9b4b-68cb-4902-832b-5907880fc427-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-krhhj\" (UID: \"3f0e9b4b-68cb-4902-832b-5907880fc427\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"
Sep 17 19:07:55.483814 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.483727    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7nkx\" (UniqueName: \"kubernetes.io/projected/3f0e9b4b-68cb-4902-832b-5907880fc427-kube-api-access-m7nkx\") pod \"openshift-state-metrics-86c5bc79d6-krhhj\" (UID: \"3f0e9b4b-68cb-4902-832b-5907880fc427\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"
Sep 17 19:07:55.483814 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.483759    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f0e9b4b-68cb-4902-832b-5907880fc427-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-krhhj\" (UID: \"3f0e9b4b-68cb-4902-832b-5907880fc427\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"
Sep 17 19:07:55.483814 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.483799    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69e4e049-b862-482f-a97a-6755cd479291-metrics-client-ca\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.484119 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.483826    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-tls\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.484119 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.483858    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-textfile\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.484119 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.483888    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-wtmp\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.484119 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.483966    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f0e9b4b-68cb-4902-832b-5907880fc427-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-krhhj\" (UID: \"3f0e9b4b-68cb-4902-832b-5907880fc427\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"
Sep 17 19:07:55.484119 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.484012    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69e4e049-b862-482f-a97a-6755cd479291-sys\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.484119 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.484049    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/69e4e049-b862-482f-a97a-6755cd479291-root\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.484531 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.484509    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f0e9b4b-68cb-4902-832b-5907880fc427-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-krhhj\" (UID: \"3f0e9b4b-68cb-4902-832b-5907880fc427\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"
Sep 17 19:07:55.487404 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.487375    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f0e9b4b-68cb-4902-832b-5907880fc427-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-krhhj\" (UID: \"3f0e9b4b-68cb-4902-832b-5907880fc427\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"
Sep 17 19:07:55.487404 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.487395    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f0e9b4b-68cb-4902-832b-5907880fc427-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-krhhj\" (UID: \"3f0e9b4b-68cb-4902-832b-5907880fc427\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"
Sep 17 19:07:55.493265 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.493238    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7nkx\" (UniqueName: \"kubernetes.io/projected/3f0e9b4b-68cb-4902-832b-5907880fc427-kube-api-access-m7nkx\") pod \"openshift-state-metrics-86c5bc79d6-krhhj\" (UID: \"3f0e9b4b-68cb-4902-832b-5907880fc427\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"
Sep 17 19:07:55.585341 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.584697    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69e4e049-b862-482f-a97a-6755cd479291-sys\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.585341 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.584752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/69e4e049-b862-482f-a97a-6755cd479291-root\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.585341 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.584781    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.585341 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.584811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-694lp\" (UniqueName: \"kubernetes.io/projected/69e4e049-b862-482f-a97a-6755cd479291-kube-api-access-694lp\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.585341 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.584844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-accelerators-collector-config\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.585341 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.584898    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69e4e049-b862-482f-a97a-6755cd479291-metrics-client-ca\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.585341 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.584924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-tls\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.585341 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.584957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-textfile\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.585341 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.584990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-wtmp\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.585341 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.585158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-wtmp\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.585341 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.585211    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69e4e049-b862-482f-a97a-6755cd479291-sys\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.585341 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.585250    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/69e4e049-b862-482f-a97a-6755cd479291-root\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.586103 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:55.585711    2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 19:07:55.586103 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:55.585775    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-tls podName:69e4e049-b862-482f-a97a-6755cd479291 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:56.085755319 +0000 UTC m=+79.913613534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-tls") pod "node-exporter-5fx8b" (UID: "69e4e049-b862-482f-a97a-6755cd479291") : secret "node-exporter-tls" not found
Sep 17 19:07:55.586346 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.586315    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69e4e049-b862-482f-a97a-6755cd479291-metrics-client-ca\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.586495 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.586479    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-accelerators-collector-config\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.586571 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.586539    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-textfile\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.587975 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.587953    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.604888 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.604862    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-694lp\" (UniqueName: \"kubernetes.io/projected/69e4e049-b862-482f-a97a-6755cd479291-kube-api-access-694lp\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:55.634750 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.634716    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"
Sep 17 19:07:55.782113 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.782071    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-66n6k"
Sep 17 19:07:55.867577 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.867483    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5468cc68cd-kq4g5" event={"ID":"fe304912-bbb0-4c00-8f44-885e054175c6","Type":"ContainerStarted","Data":"ba6face3170731503b4958513e2c1cedbe751ab9c0046713c10f3640c8c60d07"}
Sep 17 19:07:55.914167 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:55.914128    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj"]
Sep 17 19:07:55.917513 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:55.917485    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0e9b4b_68cb_4902_832b_5907880fc427.slice/crio-d34bc82e41917c98757e9382943bcedfdeb1719c0e4b2cb2fb5a31aefbe8d02b WatchSource:0}: Error finding container d34bc82e41917c98757e9382943bcedfdeb1719c0e4b2cb2fb5a31aefbe8d02b: Status 404 returned error can't find the container with id d34bc82e41917c98757e9382943bcedfdeb1719c0e4b2cb2fb5a31aefbe8d02b
Sep 17 19:07:56.090387 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.090354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-tls\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:56.092577 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.092554    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/69e4e049-b862-482f-a97a-6755cd479291-node-exporter-tls\") pod \"node-exporter-5fx8b\" (UID: \"69e4e049-b862-482f-a97a-6755cd479291\") " pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:56.270724 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.270643    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5fx8b"
Sep 17 19:07:56.281922 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:56.281894    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e4e049_b862_482f_a97a_6755cd479291.slice/crio-1b8149a27ce7ac691376ae5f12c52b71681d7cc80f9bb6bf5a6661311523ba8e WatchSource:0}: Error finding container 1b8149a27ce7ac691376ae5f12c52b71681d7cc80f9bb6bf5a6661311523ba8e: Status 404 returned error can't find the container with id 1b8149a27ce7ac691376ae5f12c52b71681d7cc80f9bb6bf5a6661311523ba8e
Sep 17 19:07:56.303620 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.303585    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:07:56.306781 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.306764    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.309704 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.309683    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 19:07:56.309823 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.309769    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 19:07:56.309823 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.309791    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 19:07:56.309936 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.309769    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 19:07:56.310108 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.310042    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 19:07:56.310108 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.310096    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 19:07:56.310277 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.310043    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 19:07:56.310277 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.310227    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 19:07:56.310431 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.310284    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-z28dl\""
Sep 17 19:07:56.310821 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.310801    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 19:07:56.323547 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.323520    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:07:56.391743 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.391707    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/08a3b90f-d6af-4732-b581-4cfef3f01112-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.391915 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.391754    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-config-volume\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.391915 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.391780    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.391915 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.391809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.391915 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.391854    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.392110 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.391935    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08a3b90f-d6af-4732-b581-4cfef3f01112-config-out\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.392110 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.391966    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.392110 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.391993    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a3b90f-d6af-4732-b581-4cfef3f01112-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.392110 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.392054    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08a3b90f-d6af-4732-b581-4cfef3f01112-tls-assets\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.392308 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.392111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.392308 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.392145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-web-config\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.392308 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.392171    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh5nt\" (UniqueName: \"kubernetes.io/projected/08a3b90f-d6af-4732-b581-4cfef3f01112-kube-api-access-kh5nt\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.392308 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.392203    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08a3b90f-d6af-4732-b581-4cfef3f01112-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.493330 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.493289    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/08a3b90f-d6af-4732-b581-4cfef3f01112-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.493530 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.493343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-config-volume\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.493530 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.493369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.493530 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.493408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.493530 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.493468    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.493530 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.493499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08a3b90f-d6af-4732-b581-4cfef3f01112-config-out\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.493530 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.493521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.493777 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.493550    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a3b90f-d6af-4732-b581-4cfef3f01112-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.493777 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.493583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08a3b90f-d6af-4732-b581-4cfef3f01112-tls-assets\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.493777 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.493608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.493777 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.493632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-web-config\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.493777 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.493654    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh5nt\" (UniqueName: \"kubernetes.io/projected/08a3b90f-d6af-4732-b581-4cfef3f01112-kube-api-access-kh5nt\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.493777 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.493682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08a3b90f-d6af-4732-b581-4cfef3f01112-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.493777 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.493736    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/08a3b90f-d6af-4732-b581-4cfef3f01112-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.494631 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.494420    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08a3b90f-d6af-4732-b581-4cfef3f01112-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.497354 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.497118    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08a3b90f-d6af-4732-b581-4cfef3f01112-config-out\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.497354 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.497199    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.497354 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:07:56.497239    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08a3b90f-d6af-4732-b581-4cfef3f01112-alertmanager-trusted-ca-bundle podName:08a3b90f-d6af-4732-b581-4cfef3f01112 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:56.997220384 +0000 UTC m=+80.825078617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/08a3b90f-d6af-4732-b581-4cfef3f01112-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "08a3b90f-d6af-4732-b581-4cfef3f01112") : configmap references non-existent config key: ca-bundle.crt
Sep 17 19:07:56.498517 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.498493    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-web-config\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.499422 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.499355    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.499573 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.499554    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08a3b90f-d6af-4732-b581-4cfef3f01112-tls-assets\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.499636 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.499588    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.500140 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.500102    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-config-volume\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.501587 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.501541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.502836 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.502809    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.503759 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.503728    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh5nt\" (UniqueName: \"kubernetes.io/projected/08a3b90f-d6af-4732-b581-4cfef3f01112-kube-api-access-kh5nt\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.881321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.881140    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj" event={"ID":"3f0e9b4b-68cb-4902-832b-5907880fc427","Type":"ContainerStarted","Data":"d987f2eaf63e6967b5e412c98e59230d1bf136cd03df5ac569c2ce48b6988669"}
Sep 17 19:07:56.881321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.881241    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj" event={"ID":"3f0e9b4b-68cb-4902-832b-5907880fc427","Type":"ContainerStarted","Data":"be1bdbd6344a0197727129cb6a30020d29dc5b11cf0b792db938dd6eab6799ee"}
Sep 17 19:07:56.881321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.881272    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj" event={"ID":"3f0e9b4b-68cb-4902-832b-5907880fc427","Type":"ContainerStarted","Data":"d34bc82e41917c98757e9382943bcedfdeb1719c0e4b2cb2fb5a31aefbe8d02b"}
Sep 17 19:07:56.884079 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.884016    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5fx8b" event={"ID":"69e4e049-b862-482f-a97a-6755cd479291","Type":"ContainerStarted","Data":"1b8149a27ce7ac691376ae5f12c52b71681d7cc80f9bb6bf5a6661311523ba8e"}
Sep 17 19:07:56.998534 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.998481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a3b90f-d6af-4732-b581-4cfef3f01112-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:56.999329 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:56.999300    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a3b90f-d6af-4732-b581-4cfef3f01112-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:57.218373 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:57.218296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:07:57.430547 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:57.430513    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:57.430711 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:57.430579    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:57.435897 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:57.435875    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:57.536189 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:57.536162    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:07:57.537507 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:07:57.537479    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a3b90f_d6af_4732_b581_4cfef3f01112.slice/crio-5b77017fe61374bf85814270a2ce4f0c58e35c47baf3f7ca89ca5f80394c788d WatchSource:0}: Error finding container 5b77017fe61374bf85814270a2ce4f0c58e35c47baf3f7ca89ca5f80394c788d: Status 404 returned error can't find the container with id 5b77017fe61374bf85814270a2ce4f0c58e35c47baf3f7ca89ca5f80394c788d
Sep 17 19:07:57.888837 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:57.888756    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a3b90f-d6af-4732-b581-4cfef3f01112","Type":"ContainerStarted","Data":"5b77017fe61374bf85814270a2ce4f0c58e35c47baf3f7ca89ca5f80394c788d"}
Sep 17 19:07:57.891214 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:57.891149    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj" event={"ID":"3f0e9b4b-68cb-4902-832b-5907880fc427","Type":"ContainerStarted","Data":"68c0d948fd976ef0ee10b98d7663cde333ea31fc2cece87fcc365e3cc6508e27"}
Sep 17 19:07:57.892993 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:57.892949    2583 generic.go:356] "Generic (PLEG): container finished" podID="69e4e049-b862-482f-a97a-6755cd479291" containerID="15ca5d1e6d5de89410ef451056dcd911a16e5dd9a75a8985e7dcd85a3b1c8df3" exitCode=0
Sep 17 19:07:57.893103 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:57.893032    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5fx8b" event={"ID":"69e4e049-b862-482f-a97a-6755cd479291","Type":"ContainerDied","Data":"15ca5d1e6d5de89410ef451056dcd911a16e5dd9a75a8985e7dcd85a3b1c8df3"}
Sep 17 19:07:57.898009 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:57.897989    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:07:57.915012 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:57.914970    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-krhhj" podStartSLOduration=1.7282016900000001 podStartE2EDuration="2.914954648s" totalImagesPullingTime="1.186752958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:57.913653442 +0000 UTC m=+81.741511727" watchObservedRunningTime="2026-09-17 19:07:57.914954648 +0000 UTC m=+81.742812888"
Sep 17 19:07:58.904600 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:58.904555    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5fx8b" event={"ID":"69e4e049-b862-482f-a97a-6755cd479291","Type":"ContainerStarted","Data":"746eedb27074e868f500812184d59af5c9cb22cb02ff2a08ce2890a5549036d8"}
Sep 17 19:07:58.905053 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:58.904655    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5fx8b" event={"ID":"69e4e049-b862-482f-a97a-6755cd479291","Type":"ContainerStarted","Data":"9827d5982296256307fbe056971cf6027f43b4ff687778aca3c0258117b38f4f"}
Sep 17 19:07:58.933613 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:58.933555    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5fx8b" podStartSLOduration=2.9053772479999997 podStartE2EDuration="3.933536838s" totalImagesPullingTime="960.378292ms" totalInitContainerRuntime="67.781298ms" isStatefulPod=true podCreationTimestamp="2026-09-17 19:07:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:58.931644601 +0000 UTC m=+82.759502838" watchObservedRunningTime="2026-09-17 19:07:58.933536838 +0000 UTC m=+82.761395073"
Sep 17 19:07:59.524342 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:59.524300    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:59.524578 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:59.524403    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:59.530673 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:59.530650    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:59.913619 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:59.913590    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:07:59.983955 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:07:59.983926    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5468cc68cd-kq4g5"]
Sep 17 19:08:00.630597 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.630561    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5668b895b4-x2sh9"]
Sep 17 19:08:00.633907 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.633882    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.637715 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.637692    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 19:08:00.637843 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.637701    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 19:08:00.638181 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.638142    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 19:08:00.638279 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.638193    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fzxqh\""
Sep 17 19:08:00.638548 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.638530    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 19:08:00.640495 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.640474    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 19:08:00.644375 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.644359    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 19:08:00.661917 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.661889    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5668b895b4-x2sh9"]
Sep 17 19:08:00.733214 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.733179    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/29ef5b94-212d-4117-a6b1-38ba1130ad41-telemeter-client-tls\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.733387 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.733223    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29ef5b94-212d-4117-a6b1-38ba1130ad41-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.733387 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.733252    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ef5b94-212d-4117-a6b1-38ba1130ad41-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.733387 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.733329    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zjtv\" (UniqueName: \"kubernetes.io/projected/29ef5b94-212d-4117-a6b1-38ba1130ad41-kube-api-access-4zjtv\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.733387 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.733361    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/29ef5b94-212d-4117-a6b1-38ba1130ad41-federate-client-tls\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.733387 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.733387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/29ef5b94-212d-4117-a6b1-38ba1130ad41-secret-telemeter-client\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.733636 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.733405    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ef5b94-212d-4117-a6b1-38ba1130ad41-serving-certs-ca-bundle\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.733636 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.733502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29ef5b94-212d-4117-a6b1-38ba1130ad41-metrics-client-ca\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.834806 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.834765    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29ef5b94-212d-4117-a6b1-38ba1130ad41-metrics-client-ca\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.834989 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.834846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/29ef5b94-212d-4117-a6b1-38ba1130ad41-telemeter-client-tls\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.834989 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.834869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29ef5b94-212d-4117-a6b1-38ba1130ad41-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.834989 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.834909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ef5b94-212d-4117-a6b1-38ba1130ad41-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.834989 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.834941    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zjtv\" (UniqueName: \"kubernetes.io/projected/29ef5b94-212d-4117-a6b1-38ba1130ad41-kube-api-access-4zjtv\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.834989 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.834964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/29ef5b94-212d-4117-a6b1-38ba1130ad41-federate-client-tls\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.835301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.835031    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/29ef5b94-212d-4117-a6b1-38ba1130ad41-secret-telemeter-client\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.835301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.835069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ef5b94-212d-4117-a6b1-38ba1130ad41-serving-certs-ca-bundle\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.835671 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.835625    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29ef5b94-212d-4117-a6b1-38ba1130ad41-metrics-client-ca\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.836019 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.835994    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ef5b94-212d-4117-a6b1-38ba1130ad41-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.836124 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.836028    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ef5b94-212d-4117-a6b1-38ba1130ad41-serving-certs-ca-bundle\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.837928 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.837883    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29ef5b94-212d-4117-a6b1-38ba1130ad41-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.838134 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.838116    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/29ef5b94-212d-4117-a6b1-38ba1130ad41-federate-client-tls\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.838254 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.838236    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/29ef5b94-212d-4117-a6b1-38ba1130ad41-telemeter-client-tls\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.838306 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.838285    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/29ef5b94-212d-4117-a6b1-38ba1130ad41-secret-telemeter-client\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.853375 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.853329    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zjtv\" (UniqueName: \"kubernetes.io/projected/29ef5b94-212d-4117-a6b1-38ba1130ad41-kube-api-access-4zjtv\") pod \"telemeter-client-5668b895b4-x2sh9\" (UID: \"29ef5b94-212d-4117-a6b1-38ba1130ad41\") " pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:00.944519 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:00.944432    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9"
Sep 17 19:08:05.935054 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:05.935021    2583 generic.go:356] "Generic (PLEG): container finished" podID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerID="619359df720cefdebc69ea7d40ebf52853f21d85abe7db24dc2fc2bb17f32766" exitCode=0
Sep 17 19:08:05.935513 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:05.935107    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a3b90f-d6af-4732-b581-4cfef3f01112","Type":"ContainerDied","Data":"619359df720cefdebc69ea7d40ebf52853f21d85abe7db24dc2fc2bb17f32766"}
Sep 17 19:08:05.939514 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:05.939490    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-lcwsh" event={"ID":"a4457e63-37ef-4ef9-8cc0-47028302cb94","Type":"ContainerStarted","Data":"26a357cad9d94ccbc9b785ae1aef4067e360b0c3d30888f44e66328b3c468314"}
Sep 17 19:08:05.939714 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:05.939688    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-lcwsh"
Sep 17 19:08:05.941025 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:05.941002    2583 patch_prober.go:28] interesting pod/downloads-6594f4c75-lcwsh container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.14:8080/\": dial tcp 10.134.0.14:8080: connect: connection refused" start-of-body=
Sep 17 19:08:05.941141 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:05.941041    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6594f4c75-lcwsh" podUID="a4457e63-37ef-4ef9-8cc0-47028302cb94" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.14:8080/\": dial tcp 10.134.0.14:8080: connect: connection refused"
Sep 17 19:08:05.979990 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:05.979928    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-lcwsh" podStartSLOduration=1.134310665 podStartE2EDuration="19.979909418s" totalImagesPullingTime="18.845598753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:05.978964069 +0000 UTC m=+89.806822311" watchObservedRunningTime="2026-09-17 19:08:05.979909418 +0000 UTC m=+89.807767654"
Sep 17 19:08:06.052645 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:06.052610    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5668b895b4-x2sh9"]
Sep 17 19:08:06.056884 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:08:06.056849    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ef5b94_212d_4117_a6b1_38ba1130ad41.slice/crio-f104ffa1fc532683ba84866dc5bd6597bda35e33b82473b8fcc1c15ada4b9ccb WatchSource:0}: Error finding container f104ffa1fc532683ba84866dc5bd6597bda35e33b82473b8fcc1c15ada4b9ccb: Status 404 returned error can't find the container with id f104ffa1fc532683ba84866dc5bd6597bda35e33b82473b8fcc1c15ada4b9ccb
Sep 17 19:08:06.787273 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:06.787231    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:08:06.948803 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:06.948631    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9" event={"ID":"29ef5b94-212d-4117-a6b1-38ba1130ad41","Type":"ContainerStarted","Data":"f104ffa1fc532683ba84866dc5bd6597bda35e33b82473b8fcc1c15ada4b9ccb"}
Sep 17 19:08:06.952904 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:06.952876    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-lcwsh"
Sep 17 19:08:08.959022 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:08.958989    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9" event={"ID":"29ef5b94-212d-4117-a6b1-38ba1130ad41","Type":"ContainerStarted","Data":"102917887caf72101ac886cb2867affe41b5b73ce7bb49876231a5ca7f09b806"}
Sep 17 19:08:08.959385 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:08.959063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9" event={"ID":"29ef5b94-212d-4117-a6b1-38ba1130ad41","Type":"ContainerStarted","Data":"f18aebf2283daf49d7fa91bcbca69ca3dabbc611cd40550dc4c50793e68aba43"}
Sep 17 19:08:08.961851 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:08.961821    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a3b90f-d6af-4732-b581-4cfef3f01112","Type":"ContainerStarted","Data":"6e4a82bf11ade325771062efaa53acd00333d9a4390ea200cb03ec85b91247bb"}
Sep 17 19:08:08.961954 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:08.961900    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a3b90f-d6af-4732-b581-4cfef3f01112","Type":"ContainerStarted","Data":"676dd0a64537582a42e67827fef5f638f417dc4af1e4862d402b3bd31ff1eb0d"}
Sep 17 19:08:09.829571 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:09.829540    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c588ccb4c-d45nx"
Sep 17 19:08:09.969637 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:09.969596    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9" event={"ID":"29ef5b94-212d-4117-a6b1-38ba1130ad41","Type":"ContainerStarted","Data":"72f195fb43ce5f39d5f343565fb4ab722c4f148c1dd1e9841d9034999ff3d98f"}
Sep 17 19:08:09.973321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:09.973293    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a3b90f-d6af-4732-b581-4cfef3f01112","Type":"ContainerStarted","Data":"ccbe3c3b0f813443dd871dea8759a1003bb4b5beae06c793a6bb90069df4ceda"}
Sep 17 19:08:09.973473 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:09.973360    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a3b90f-d6af-4732-b581-4cfef3f01112","Type":"ContainerStarted","Data":"73b3d7e3255a2a8213ce82ea6e4dcd4adea8bb2eedba0fb7ea3e366a0f44d73e"}
Sep 17 19:08:09.973473 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:09.973396    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a3b90f-d6af-4732-b581-4cfef3f01112","Type":"ContainerStarted","Data":"63b24679729fd33c93d3f4565fe718d9acdb3ff065d986658e33343e85a3012a"}
Sep 17 19:08:10.018312 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.018250    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5668b895b4-x2sh9" podStartSLOduration=7.434541543 podStartE2EDuration="10.01822681s" totalImagesPullingTime="2.583685267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:10.016302554 +0000 UTC m=+93.844160812" watchObservedRunningTime="2026-09-17 19:08:10.01822681 +0000 UTC m=+93.846085061"
Sep 17 19:08:10.174825 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.174748    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5797768c74-j2tqx"]
Sep 17 19:08:10.201159 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.201127    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5797768c74-j2tqx"]
Sep 17 19:08:10.201330 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.201253    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.329248 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.329209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-oauth-config\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.329248 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.329245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-config\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.329503 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.329271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-service-ca\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.329503 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.329367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxnk9\" (UniqueName: \"kubernetes.io/projected/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-kube-api-access-vxnk9\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.329503 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.329436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-serving-cert\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.329503 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.329489    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-oauth-serving-cert\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.329700 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.329530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-trusted-ca-bundle\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.430787 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.430710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-serving-cert\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.430787 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.430755    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-oauth-serving-cert\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.431003 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.430795    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-trusted-ca-bundle\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.431003 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.430853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-oauth-config\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.431003 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.430879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-config\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.431003 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.430910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-service-ca\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.431003 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.430957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxnk9\" (UniqueName: \"kubernetes.io/projected/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-kube-api-access-vxnk9\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.431890 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.431865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-oauth-serving-cert\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.431990 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.431921    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-config\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.432088 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.432062    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-trusted-ca-bundle\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.432138 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.432082    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-service-ca\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.433662 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.433639    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-oauth-config\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.433775 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.433756    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-serving-cert\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.440502 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.440478    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxnk9\" (UniqueName: \"kubernetes.io/projected/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-kube-api-access-vxnk9\") pod \"console-5797768c74-j2tqx\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") " pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.513397 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.513357    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:10.825737 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.825710    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5797768c74-j2tqx"]
Sep 17 19:08:10.941543 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:08:10.941497    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1893f0_2284_4b8b_869d_37bf5c0b1f17.slice/crio-0ad50390bcb1a034827c4d789b4acb26d3e26b8e0dd83aca74fff47333dc9308 WatchSource:0}: Error finding container 0ad50390bcb1a034827c4d789b4acb26d3e26b8e0dd83aca74fff47333dc9308: Status 404 returned error can't find the container with id 0ad50390bcb1a034827c4d789b4acb26d3e26b8e0dd83aca74fff47333dc9308
Sep 17 19:08:10.978664 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:10.978634    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5797768c74-j2tqx" event={"ID":"2a1893f0-2284-4b8b-869d-37bf5c0b1f17","Type":"ContainerStarted","Data":"0ad50390bcb1a034827c4d789b4acb26d3e26b8e0dd83aca74fff47333dc9308"}
Sep 17 19:08:11.797519 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:11.797436    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-648597d777-lfb4m" podUID="74441563-19c2-4fe5-8802-9db0426c73f3" containerName="registry" containerID="cri-o://a243262146953cc04536e5fbb4e2619933ecd49f31f5a28712cc9983f55a3a0f" gracePeriod=30
Sep 17 19:08:11.988689 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:11.988656    2583 generic.go:356] "Generic (PLEG): container finished" podID="74441563-19c2-4fe5-8802-9db0426c73f3" containerID="a243262146953cc04536e5fbb4e2619933ecd49f31f5a28712cc9983f55a3a0f" exitCode=0
Sep 17 19:08:11.989081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:11.988760    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648597d777-lfb4m" event={"ID":"74441563-19c2-4fe5-8802-9db0426c73f3","Type":"ContainerDied","Data":"a243262146953cc04536e5fbb4e2619933ecd49f31f5a28712cc9983f55a3a0f"}
Sep 17 19:08:11.990846 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:11.990820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5797768c74-j2tqx" event={"ID":"2a1893f0-2284-4b8b-869d-37bf5c0b1f17","Type":"ContainerStarted","Data":"156c6db44a416e98f4093714a64c57e476d270632337559ec08457880613cb33"}
Sep 17 19:08:11.996324 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:11.996276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a3b90f-d6af-4732-b581-4cfef3f01112","Type":"ContainerStarted","Data":"e4e98ff232ee7834cdb842e404f1577603a94ae69673ce38544a6f9612dc2566"}
Sep 17 19:08:12.014098 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.014044    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5797768c74-j2tqx" podStartSLOduration=2.014025714 podStartE2EDuration="2.014025714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:12.012012632 +0000 UTC m=+95.839870867" watchObservedRunningTime="2026-09-17 19:08:12.014025714 +0000 UTC m=+95.841883949"
Sep 17 19:08:12.043703 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.043642    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.490281094 podStartE2EDuration="16.04362127s" totalImagesPullingTime="12.536418061s" totalInitContainerRuntime="16.922115ms" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:12.040878812 +0000 UTC m=+95.868737071" watchObservedRunningTime="2026-09-17 19:08:12.04362127 +0000 UTC m=+95.871479506"
Sep 17 19:08:12.099069 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.099038    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:08:12.249464 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.249413    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74441563-19c2-4fe5-8802-9db0426c73f3-ca-trust-extracted\") pod \"74441563-19c2-4fe5-8802-9db0426c73f3\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") "
Sep 17 19:08:12.249711 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.249493    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74441563-19c2-4fe5-8802-9db0426c73f3-image-registry-private-configuration\") pod \"74441563-19c2-4fe5-8802-9db0426c73f3\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") "
Sep 17 19:08:12.249711 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.249521    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74441563-19c2-4fe5-8802-9db0426c73f3-installation-pull-secrets\") pod \"74441563-19c2-4fe5-8802-9db0426c73f3\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") "
Sep 17 19:08:12.249711 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.249544    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74441563-19c2-4fe5-8802-9db0426c73f3-trusted-ca\") pod \"74441563-19c2-4fe5-8802-9db0426c73f3\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") "
Sep 17 19:08:12.249711 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.249574    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2vk2\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-kube-api-access-k2vk2\") pod \"74441563-19c2-4fe5-8802-9db0426c73f3\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") "
Sep 17 19:08:12.249711 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.249609    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74441563-19c2-4fe5-8802-9db0426c73f3-registry-certificates\") pod \"74441563-19c2-4fe5-8802-9db0426c73f3\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") "
Sep 17 19:08:12.249711 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.249633    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-bound-sa-token\") pod \"74441563-19c2-4fe5-8802-9db0426c73f3\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") "
Sep 17 19:08:12.249711 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.249675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls\") pod \"74441563-19c2-4fe5-8802-9db0426c73f3\" (UID: \"74441563-19c2-4fe5-8802-9db0426c73f3\") "
Sep 17 19:08:12.250096 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.249973    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74441563-19c2-4fe5-8802-9db0426c73f3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "74441563-19c2-4fe5-8802-9db0426c73f3" (UID: "74441563-19c2-4fe5-8802-9db0426c73f3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:12.250482 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.250285    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74441563-19c2-4fe5-8802-9db0426c73f3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "74441563-19c2-4fe5-8802-9db0426c73f3" (UID: "74441563-19c2-4fe5-8802-9db0426c73f3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:12.252466 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.252417    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "74441563-19c2-4fe5-8802-9db0426c73f3" (UID: "74441563-19c2-4fe5-8802-9db0426c73f3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:12.253457 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.253409    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74441563-19c2-4fe5-8802-9db0426c73f3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "74441563-19c2-4fe5-8802-9db0426c73f3" (UID: "74441563-19c2-4fe5-8802-9db0426c73f3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:12.253557 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.253499    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74441563-19c2-4fe5-8802-9db0426c73f3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "74441563-19c2-4fe5-8802-9db0426c73f3" (UID: "74441563-19c2-4fe5-8802-9db0426c73f3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:12.253616 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.253559    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "74441563-19c2-4fe5-8802-9db0426c73f3" (UID: "74441563-19c2-4fe5-8802-9db0426c73f3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:12.253671 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.253612    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-kube-api-access-k2vk2" (OuterVolumeSpecName: "kube-api-access-k2vk2") pod "74441563-19c2-4fe5-8802-9db0426c73f3" (UID: "74441563-19c2-4fe5-8802-9db0426c73f3"). InnerVolumeSpecName "kube-api-access-k2vk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:12.260551 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.260494    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74441563-19c2-4fe5-8802-9db0426c73f3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "74441563-19c2-4fe5-8802-9db0426c73f3" (UID: "74441563-19c2-4fe5-8802-9db0426c73f3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:08:12.350685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.350612    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/74441563-19c2-4fe5-8802-9db0426c73f3-ca-trust-extracted\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.350685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.350645    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/74441563-19c2-4fe5-8802-9db0426c73f3-image-registry-private-configuration\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.350685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.350658    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/74441563-19c2-4fe5-8802-9db0426c73f3-installation-pull-secrets\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.350685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.350668    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/74441563-19c2-4fe5-8802-9db0426c73f3-trusted-ca\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.350685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.350677    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2vk2\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-kube-api-access-k2vk2\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.350685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.350685    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/74441563-19c2-4fe5-8802-9db0426c73f3-registry-certificates\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.351030 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.350693    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-bound-sa-token\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:12.351030 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:12.350701    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/74441563-19c2-4fe5-8802-9db0426c73f3-registry-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:13.002797 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:13.002767    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648597d777-lfb4m"
Sep 17 19:08:13.003330 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:13.002807    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648597d777-lfb4m" event={"ID":"74441563-19c2-4fe5-8802-9db0426c73f3","Type":"ContainerDied","Data":"3b3890d758e30e5b6ae85e48fb34fcfbcd067c433a1d9c9555d6f814c6f3ab56"}
Sep 17 19:08:13.003330 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:13.002882    2583 scope.go:117] "RemoveContainer" containerID="a243262146953cc04536e5fbb4e2619933ecd49f31f5a28712cc9983f55a3a0f"
Sep 17 19:08:13.021976 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:13.021953    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-648597d777-lfb4m"]
Sep 17 19:08:13.029817 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:13.029791    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-648597d777-lfb4m"]
Sep 17 19:08:14.457097 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:14.457060    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74441563-19c2-4fe5-8802-9db0426c73f3" path="/var/lib/kubelet/pods/74441563-19c2-4fe5-8802-9db0426c73f3/volumes"
Sep 17 19:08:20.514207 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:20.514163    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:20.514824 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:20.514243    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:20.518751 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:20.518729    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:21.037780 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:21.037748    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:08:21.093868 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:21.093835    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54bdd75bb4-fd24d"]
Sep 17 19:08:25.934004 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:25.933965    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5468cc68cd-kq4g5" podUID="fe304912-bbb0-4c00-8f44-885e054175c6" containerName="console" containerID="cri-o://ba6face3170731503b4958513e2c1cedbe751ab9c0046713c10f3640c8c60d07" gracePeriod=15
Sep 17 19:08:26.059119 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.059088    2583 generic.go:356] "Generic (PLEG): container finished" podID="fe304912-bbb0-4c00-8f44-885e054175c6" containerID="ba6face3170731503b4958513e2c1cedbe751ab9c0046713c10f3640c8c60d07" exitCode=0
Sep 17 19:08:26.059277 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.059157    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5468cc68cd-kq4g5" event={"ID":"fe304912-bbb0-4c00-8f44-885e054175c6","Type":"ContainerDied","Data":"ba6face3170731503b4958513e2c1cedbe751ab9c0046713c10f3640c8c60d07"}
Sep 17 19:08:26.197798 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.197738    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:08:26.265395 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.265371    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6brc\" (UniqueName: \"kubernetes.io/projected/fe304912-bbb0-4c00-8f44-885e054175c6-kube-api-access-s6brc\") pod \"fe304912-bbb0-4c00-8f44-885e054175c6\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") "
Sep 17 19:08:26.265574 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.265410    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-service-ca\") pod \"fe304912-bbb0-4c00-8f44-885e054175c6\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") "
Sep 17 19:08:26.265574 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.265432    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-oauth-serving-cert\") pod \"fe304912-bbb0-4c00-8f44-885e054175c6\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") "
Sep 17 19:08:26.265574 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.265489    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe304912-bbb0-4c00-8f44-885e054175c6-console-oauth-config\") pod \"fe304912-bbb0-4c00-8f44-885e054175c6\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") "
Sep 17 19:08:26.265574 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.265526    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe304912-bbb0-4c00-8f44-885e054175c6-console-serving-cert\") pod \"fe304912-bbb0-4c00-8f44-885e054175c6\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") "
Sep 17 19:08:26.265775 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.265675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-console-config\") pod \"fe304912-bbb0-4c00-8f44-885e054175c6\" (UID: \"fe304912-bbb0-4c00-8f44-885e054175c6\") "
Sep 17 19:08:26.265878 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.265854    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-service-ca" (OuterVolumeSpecName: "service-ca") pod "fe304912-bbb0-4c00-8f44-885e054175c6" (UID: "fe304912-bbb0-4c00-8f44-885e054175c6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:26.265939 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.265920    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fe304912-bbb0-4c00-8f44-885e054175c6" (UID: "fe304912-bbb0-4c00-8f44-885e054175c6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:26.266001 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.265973    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-service-ca\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:26.266001 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.265989    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-console-config" (OuterVolumeSpecName: "console-config") pod "fe304912-bbb0-4c00-8f44-885e054175c6" (UID: "fe304912-bbb0-4c00-8f44-885e054175c6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:26.267824 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.267794    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe304912-bbb0-4c00-8f44-885e054175c6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fe304912-bbb0-4c00-8f44-885e054175c6" (UID: "fe304912-bbb0-4c00-8f44-885e054175c6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:26.267824 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.267818    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe304912-bbb0-4c00-8f44-885e054175c6-kube-api-access-s6brc" (OuterVolumeSpecName: "kube-api-access-s6brc") pod "fe304912-bbb0-4c00-8f44-885e054175c6" (UID: "fe304912-bbb0-4c00-8f44-885e054175c6"). InnerVolumeSpecName "kube-api-access-s6brc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:26.267971 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.267877    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe304912-bbb0-4c00-8f44-885e054175c6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fe304912-bbb0-4c00-8f44-885e054175c6" (UID: "fe304912-bbb0-4c00-8f44-885e054175c6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:26.366339 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.366303    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6brc\" (UniqueName: \"kubernetes.io/projected/fe304912-bbb0-4c00-8f44-885e054175c6-kube-api-access-s6brc\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:26.366339 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.366330    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-oauth-serving-cert\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:26.366339 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.366339    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe304912-bbb0-4c00-8f44-885e054175c6-console-oauth-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:26.366589 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.366348    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe304912-bbb0-4c00-8f44-885e054175c6-console-serving-cert\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:26.366589 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:26.366360    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe304912-bbb0-4c00-8f44-885e054175c6-console-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:27.063930 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:27.063900    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5468cc68cd-kq4g5"
Sep 17 19:08:27.064371 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:27.063901    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5468cc68cd-kq4g5" event={"ID":"fe304912-bbb0-4c00-8f44-885e054175c6","Type":"ContainerDied","Data":"53ead9535703d97197db48a1b3f96237d4f4e9aee82482562c880528d7912bd0"}
Sep 17 19:08:27.064371 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:27.064036    2583 scope.go:117] "RemoveContainer" containerID="ba6face3170731503b4958513e2c1cedbe751ab9c0046713c10f3640c8c60d07"
Sep 17 19:08:27.081795 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:27.081773    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5468cc68cd-kq4g5"]
Sep 17 19:08:27.084164 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:27.084144    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5468cc68cd-kq4g5"]
Sep 17 19:08:28.450930 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:28.450903    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe304912-bbb0-4c00-8f44-885e054175c6" path="/var/lib/kubelet/pods/fe304912-bbb0-4c00-8f44-885e054175c6/volumes"
Sep 17 19:08:46.113435 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.113391    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54bdd75bb4-fd24d" podUID="5c61d97a-58be-4e9f-8893-025eee26298e" containerName="console" containerID="cri-o://6cb92e997305cfd6a081d44478890aa1a5fcaa585298566cf50414fbb3c93403" gracePeriod=15
Sep 17 19:08:46.393562 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.393490    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:08:46.537763 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.537725    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c61d97a-58be-4e9f-8893-025eee26298e-console-oauth-config\") pod \"5c61d97a-58be-4e9f-8893-025eee26298e\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") "
Sep 17 19:08:46.537763 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.537763    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnd9t\" (UniqueName: \"kubernetes.io/projected/5c61d97a-58be-4e9f-8893-025eee26298e-kube-api-access-nnd9t\") pod \"5c61d97a-58be-4e9f-8893-025eee26298e\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") "
Sep 17 19:08:46.538000 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.537788    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-console-config\") pod \"5c61d97a-58be-4e9f-8893-025eee26298e\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") "
Sep 17 19:08:46.538000 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.537846    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-service-ca\") pod \"5c61d97a-58be-4e9f-8893-025eee26298e\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") "
Sep 17 19:08:46.538000 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.537886    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c61d97a-58be-4e9f-8893-025eee26298e-console-serving-cert\") pod \"5c61d97a-58be-4e9f-8893-025eee26298e\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") "
Sep 17 19:08:46.538000 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.537917    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-oauth-serving-cert\") pod \"5c61d97a-58be-4e9f-8893-025eee26298e\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") "
Sep 17 19:08:46.538000 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.537952    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-trusted-ca-bundle\") pod \"5c61d97a-58be-4e9f-8893-025eee26298e\" (UID: \"5c61d97a-58be-4e9f-8893-025eee26298e\") "
Sep 17 19:08:46.538323 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.538289    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-service-ca" (OuterVolumeSpecName: "service-ca") pod "5c61d97a-58be-4e9f-8893-025eee26298e" (UID: "5c61d97a-58be-4e9f-8893-025eee26298e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:46.538323 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.538288    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-console-config" (OuterVolumeSpecName: "console-config") pod "5c61d97a-58be-4e9f-8893-025eee26298e" (UID: "5c61d97a-58be-4e9f-8893-025eee26298e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:46.538705 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.538680    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5c61d97a-58be-4e9f-8893-025eee26298e" (UID: "5c61d97a-58be-4e9f-8893-025eee26298e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:46.538872 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.538845    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5c61d97a-58be-4e9f-8893-025eee26298e" (UID: "5c61d97a-58be-4e9f-8893-025eee26298e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:46.540056 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.540026    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c61d97a-58be-4e9f-8893-025eee26298e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5c61d97a-58be-4e9f-8893-025eee26298e" (UID: "5c61d97a-58be-4e9f-8893-025eee26298e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:46.540155 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.540115    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c61d97a-58be-4e9f-8893-025eee26298e-kube-api-access-nnd9t" (OuterVolumeSpecName: "kube-api-access-nnd9t") pod "5c61d97a-58be-4e9f-8893-025eee26298e" (UID: "5c61d97a-58be-4e9f-8893-025eee26298e"). InnerVolumeSpecName "kube-api-access-nnd9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:46.540155 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.540132    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c61d97a-58be-4e9f-8893-025eee26298e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5c61d97a-58be-4e9f-8893-025eee26298e" (UID: "5c61d97a-58be-4e9f-8893-025eee26298e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:46.639227 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.639192    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-console-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:46.639227 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.639219    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-service-ca\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:46.639227 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.639228    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c61d97a-58be-4e9f-8893-025eee26298e-console-serving-cert\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:46.639227 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.639238    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-oauth-serving-cert\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:46.639509 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.639247    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c61d97a-58be-4e9f-8893-025eee26298e-trusted-ca-bundle\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:46.639509 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.639255    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c61d97a-58be-4e9f-8893-025eee26298e-console-oauth-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:46.639509 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:46.639263    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnd9t\" (UniqueName: \"kubernetes.io/projected/5c61d97a-58be-4e9f-8893-025eee26298e-kube-api-access-nnd9t\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:08:47.139958 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:47.139926    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c61d97a-58be-4e9f-8893-025eee26298e" containerID="6cb92e997305cfd6a081d44478890aa1a5fcaa585298566cf50414fbb3c93403" exitCode=0
Sep 17 19:08:47.140354 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:47.139967    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54bdd75bb4-fd24d"
Sep 17 19:08:47.140354 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:47.140018    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54bdd75bb4-fd24d" event={"ID":"5c61d97a-58be-4e9f-8893-025eee26298e","Type":"ContainerDied","Data":"6cb92e997305cfd6a081d44478890aa1a5fcaa585298566cf50414fbb3c93403"}
Sep 17 19:08:47.140354 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:47.140062    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54bdd75bb4-fd24d" event={"ID":"5c61d97a-58be-4e9f-8893-025eee26298e","Type":"ContainerDied","Data":"7088534460bddf92713dde03dbefdb929b860bd556bebd056dc5a46e0e1974a3"}
Sep 17 19:08:47.140354 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:47.140084    2583 scope.go:117] "RemoveContainer" containerID="6cb92e997305cfd6a081d44478890aa1a5fcaa585298566cf50414fbb3c93403"
Sep 17 19:08:47.148474 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:47.148459    2583 scope.go:117] "RemoveContainer" containerID="6cb92e997305cfd6a081d44478890aa1a5fcaa585298566cf50414fbb3c93403"
Sep 17 19:08:47.148765 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:08:47.148747    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb92e997305cfd6a081d44478890aa1a5fcaa585298566cf50414fbb3c93403\": container with ID starting with 6cb92e997305cfd6a081d44478890aa1a5fcaa585298566cf50414fbb3c93403 not found: ID does not exist" containerID="6cb92e997305cfd6a081d44478890aa1a5fcaa585298566cf50414fbb3c93403"
Sep 17 19:08:47.148804 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:47.148775    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb92e997305cfd6a081d44478890aa1a5fcaa585298566cf50414fbb3c93403"} err="failed to get container status \"6cb92e997305cfd6a081d44478890aa1a5fcaa585298566cf50414fbb3c93403\": rpc error: code = NotFound desc = could not find container \"6cb92e997305cfd6a081d44478890aa1a5fcaa585298566cf50414fbb3c93403\": container with ID starting with 6cb92e997305cfd6a081d44478890aa1a5fcaa585298566cf50414fbb3c93403 not found: ID does not exist"
Sep 17 19:08:47.160974 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:47.160953    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54bdd75bb4-fd24d"]
Sep 17 19:08:47.164884 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:47.164865    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54bdd75bb4-fd24d"]
Sep 17 19:08:48.451403 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:08:48.451367    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c61d97a-58be-4e9f-8893-025eee26298e" path="/var/lib/kubelet/pods/5c61d97a-58be-4e9f-8893-025eee26298e/volumes"
Sep 17 19:09:11.453177 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.453145    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c889d49bb-fxp5r"]
Sep 17 19:09:11.453773 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.453574    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c61d97a-58be-4e9f-8893-025eee26298e" containerName="console"
Sep 17 19:09:11.453773 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.453591    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c61d97a-58be-4e9f-8893-025eee26298e" containerName="console"
Sep 17 19:09:11.453773 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.453603    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74441563-19c2-4fe5-8802-9db0426c73f3" containerName="registry"
Sep 17 19:09:11.453773 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.453613    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="74441563-19c2-4fe5-8802-9db0426c73f3" containerName="registry"
Sep 17 19:09:11.453773 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.453628    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe304912-bbb0-4c00-8f44-885e054175c6" containerName="console"
Sep 17 19:09:11.453773 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.453637    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe304912-bbb0-4c00-8f44-885e054175c6" containerName="console"
Sep 17 19:09:11.453773 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.453710    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c61d97a-58be-4e9f-8893-025eee26298e" containerName="console"
Sep 17 19:09:11.453773 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.453720    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="74441563-19c2-4fe5-8802-9db0426c73f3" containerName="registry"
Sep 17 19:09:11.453773 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.453730    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe304912-bbb0-4c00-8f44-885e054175c6" containerName="console"
Sep 17 19:09:11.456830 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.456809    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.466755 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.466735    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c889d49bb-fxp5r"]
Sep 17 19:09:11.641929 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.641889    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-serving-cert\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.642105 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.641942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-config\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.642105 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.642001    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-oauth-serving-cert\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.642105 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.642078    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-oauth-config\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.642208 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.642145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-trusted-ca-bundle\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.642208 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.642198    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78lk\" (UniqueName: \"kubernetes.io/projected/9f08e542-7c5f-459d-970f-8cc3a34b3546-kube-api-access-m78lk\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.642276 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.642230    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-service-ca\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.742913 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.742843    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-serving-cert\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.742913 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.742882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-config\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.742913 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.742901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-oauth-serving-cert\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.743095 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.742924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-oauth-config\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.743095 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.742960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-trusted-ca-bundle\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.743095 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.742980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78lk\" (UniqueName: \"kubernetes.io/projected/9f08e542-7c5f-459d-970f-8cc3a34b3546-kube-api-access-m78lk\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.743095 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.742998    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-service-ca\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.743685 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.743655    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-oauth-serving-cert\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.743802 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.743748    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-config\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.743865 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.743807    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-trusted-ca-bundle\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.743865 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.743816    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-service-ca\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.745292 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.745265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-oauth-config\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.745373 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.745362    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-serving-cert\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.752266 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.752249    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78lk\" (UniqueName: \"kubernetes.io/projected/9f08e542-7c5f-459d-970f-8cc3a34b3546-kube-api-access-m78lk\") pod \"console-5c889d49bb-fxp5r\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") " pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:11.765208 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:11.765186    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:12.038638 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:12.038597    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c889d49bb-fxp5r"]
Sep 17 19:09:12.041399 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:09:12.041374    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f08e542_7c5f_459d_970f_8cc3a34b3546.slice/crio-0f215529cfe079c3622a9f3c79d28c16110edd1e1e2c084367e45cc619a09478 WatchSource:0}: Error finding container 0f215529cfe079c3622a9f3c79d28c16110edd1e1e2c084367e45cc619a09478: Status 404 returned error can't find the container with id 0f215529cfe079c3622a9f3c79d28c16110edd1e1e2c084367e45cc619a09478
Sep 17 19:09:12.237341 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:12.237305    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c889d49bb-fxp5r" event={"ID":"9f08e542-7c5f-459d-970f-8cc3a34b3546","Type":"ContainerStarted","Data":"76eb57e2c475fc35cc22e728269f6ce6e518b5ec7131ffe3ab6a57c147f9635d"}
Sep 17 19:09:12.237515 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:12.237357    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c889d49bb-fxp5r" event={"ID":"9f08e542-7c5f-459d-970f-8cc3a34b3546","Type":"ContainerStarted","Data":"0f215529cfe079c3622a9f3c79d28c16110edd1e1e2c084367e45cc619a09478"}
Sep 17 19:09:12.262468 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:12.260735    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c889d49bb-fxp5r" podStartSLOduration=1.260716596 podStartE2EDuration="1.260716596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:12.257981685 +0000 UTC m=+156.085839918" watchObservedRunningTime="2026-09-17 19:09:12.260716596 +0000 UTC m=+156.088574834"
Sep 17 19:09:15.721665 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:15.721632    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:09:15.722017 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:15.721909    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="alertmanager" containerID="cri-o://676dd0a64537582a42e67827fef5f638f417dc4af1e4862d402b3bd31ff1eb0d" gracePeriod=120
Sep 17 19:09:15.722017 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:15.721968    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="kube-rbac-proxy-metric" containerID="cri-o://ccbe3c3b0f813443dd871dea8759a1003bb4b5beae06c793a6bb90069df4ceda" gracePeriod=120
Sep 17 19:09:15.722017 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:15.721994    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="prom-label-proxy" containerID="cri-o://e4e98ff232ee7834cdb842e404f1577603a94ae69673ce38544a6f9612dc2566" gracePeriod=120
Sep 17 19:09:15.722017 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:15.722002    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="kube-rbac-proxy-web" containerID="cri-o://63b24679729fd33c93d3f4565fe718d9acdb3ff065d986658e33343e85a3012a" gracePeriod=120
Sep 17 19:09:15.722204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:15.722022    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="kube-rbac-proxy" containerID="cri-o://73b3d7e3255a2a8213ce82ea6e4dcd4adea8bb2eedba0fb7ea3e366a0f44d73e" gracePeriod=120
Sep 17 19:09:15.722204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:15.722053    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="config-reloader" containerID="cri-o://6e4a82bf11ade325771062efaa53acd00333d9a4390ea200cb03ec85b91247bb" gracePeriod=120
Sep 17 19:09:16.254031 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.253998    2583 generic.go:356] "Generic (PLEG): container finished" podID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerID="e4e98ff232ee7834cdb842e404f1577603a94ae69673ce38544a6f9612dc2566" exitCode=0
Sep 17 19:09:16.254031 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.254021    2583 generic.go:356] "Generic (PLEG): container finished" podID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerID="ccbe3c3b0f813443dd871dea8759a1003bb4b5beae06c793a6bb90069df4ceda" exitCode=0
Sep 17 19:09:16.254031 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.254028    2583 generic.go:356] "Generic (PLEG): container finished" podID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerID="73b3d7e3255a2a8213ce82ea6e4dcd4adea8bb2eedba0fb7ea3e366a0f44d73e" exitCode=0
Sep 17 19:09:16.254031 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.254034    2583 generic.go:356] "Generic (PLEG): container finished" podID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerID="6e4a82bf11ade325771062efaa53acd00333d9a4390ea200cb03ec85b91247bb" exitCode=0
Sep 17 19:09:16.254031 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.254039    2583 generic.go:356] "Generic (PLEG): container finished" podID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerID="676dd0a64537582a42e67827fef5f638f417dc4af1e4862d402b3bd31ff1eb0d" exitCode=0
Sep 17 19:09:16.254300 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.254071    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a3b90f-d6af-4732-b581-4cfef3f01112","Type":"ContainerDied","Data":"e4e98ff232ee7834cdb842e404f1577603a94ae69673ce38544a6f9612dc2566"}
Sep 17 19:09:16.254300 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.254134    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a3b90f-d6af-4732-b581-4cfef3f01112","Type":"ContainerDied","Data":"ccbe3c3b0f813443dd871dea8759a1003bb4b5beae06c793a6bb90069df4ceda"}
Sep 17 19:09:16.254300 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.254159    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a3b90f-d6af-4732-b581-4cfef3f01112","Type":"ContainerDied","Data":"73b3d7e3255a2a8213ce82ea6e4dcd4adea8bb2eedba0fb7ea3e366a0f44d73e"}
Sep 17 19:09:16.254300 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.254184    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a3b90f-d6af-4732-b581-4cfef3f01112","Type":"ContainerDied","Data":"6e4a82bf11ade325771062efaa53acd00333d9a4390ea200cb03ec85b91247bb"}
Sep 17 19:09:16.254300 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.254206    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a3b90f-d6af-4732-b581-4cfef3f01112","Type":"ContainerDied","Data":"676dd0a64537582a42e67827fef5f638f417dc4af1e4862d402b3bd31ff1eb0d"}
Sep 17 19:09:16.954097 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.954074    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:16.982326 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.982297    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08a3b90f-d6af-4732-b581-4cfef3f01112-tls-assets\") pod \"08a3b90f-d6af-4732-b581-4cfef3f01112\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") "
Sep 17 19:09:16.982486 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.982344    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08a3b90f-d6af-4732-b581-4cfef3f01112-metrics-client-ca\") pod \"08a3b90f-d6af-4732-b581-4cfef3f01112\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") "
Sep 17 19:09:16.982486 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.982372    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/08a3b90f-d6af-4732-b581-4cfef3f01112-alertmanager-main-db\") pod \"08a3b90f-d6af-4732-b581-4cfef3f01112\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") "
Sep 17 19:09:16.982486 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.982397    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-main-tls\") pod \"08a3b90f-d6af-4732-b581-4cfef3f01112\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") "
Sep 17 19:09:16.982644 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.982490    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-config-volume\") pod \"08a3b90f-d6af-4732-b581-4cfef3f01112\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") "
Sep 17 19:09:16.982644 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.982541    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy-web\") pod \"08a3b90f-d6af-4732-b581-4cfef3f01112\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") "
Sep 17 19:09:16.982644 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.982572    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy\") pod \"08a3b90f-d6af-4732-b581-4cfef3f01112\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") "
Sep 17 19:09:16.982644 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.982600    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy-metric\") pod \"08a3b90f-d6af-4732-b581-4cfef3f01112\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") "
Sep 17 19:09:16.982644 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.982635    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-web-config\") pod \"08a3b90f-d6af-4732-b581-4cfef3f01112\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") "
Sep 17 19:09:16.982879 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.982677    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-cluster-tls-config\") pod \"08a3b90f-d6af-4732-b581-4cfef3f01112\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") "
Sep 17 19:09:16.982879 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.982707    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a3b90f-d6af-4732-b581-4cfef3f01112-alertmanager-trusted-ca-bundle\") pod \"08a3b90f-d6af-4732-b581-4cfef3f01112\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") "
Sep 17 19:09:16.982879 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.982741    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08a3b90f-d6af-4732-b581-4cfef3f01112-config-out\") pod \"08a3b90f-d6af-4732-b581-4cfef3f01112\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") "
Sep 17 19:09:16.982879 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.982791    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh5nt\" (UniqueName: \"kubernetes.io/projected/08a3b90f-d6af-4732-b581-4cfef3f01112-kube-api-access-kh5nt\") pod \"08a3b90f-d6af-4732-b581-4cfef3f01112\" (UID: \"08a3b90f-d6af-4732-b581-4cfef3f01112\") "
Sep 17 19:09:16.983895 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.982672    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a3b90f-d6af-4732-b581-4cfef3f01112-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "08a3b90f-d6af-4732-b581-4cfef3f01112" (UID: "08a3b90f-d6af-4732-b581-4cfef3f01112"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:09:16.983895 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.982727    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a3b90f-d6af-4732-b581-4cfef3f01112-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "08a3b90f-d6af-4732-b581-4cfef3f01112" (UID: "08a3b90f-d6af-4732-b581-4cfef3f01112"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:09:16.984069 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.983860    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a3b90f-d6af-4732-b581-4cfef3f01112-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "08a3b90f-d6af-4732-b581-4cfef3f01112" (UID: "08a3b90f-d6af-4732-b581-4cfef3f01112"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:09:16.987555 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.985382    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "08a3b90f-d6af-4732-b581-4cfef3f01112" (UID: "08a3b90f-d6af-4732-b581-4cfef3f01112"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:09:16.988344 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.988298    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a3b90f-d6af-4732-b581-4cfef3f01112-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "08a3b90f-d6af-4732-b581-4cfef3f01112" (UID: "08a3b90f-d6af-4732-b581-4cfef3f01112"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:09:16.988591 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.988556    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "08a3b90f-d6af-4732-b581-4cfef3f01112" (UID: "08a3b90f-d6af-4732-b581-4cfef3f01112"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:09:16.989408 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.989383    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a3b90f-d6af-4732-b581-4cfef3f01112-kube-api-access-kh5nt" (OuterVolumeSpecName: "kube-api-access-kh5nt") pod "08a3b90f-d6af-4732-b581-4cfef3f01112" (UID: "08a3b90f-d6af-4732-b581-4cfef3f01112"). InnerVolumeSpecName "kube-api-access-kh5nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:09:16.990312 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.990271    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a3b90f-d6af-4732-b581-4cfef3f01112-config-out" (OuterVolumeSpecName: "config-out") pod "08a3b90f-d6af-4732-b581-4cfef3f01112" (UID: "08a3b90f-d6af-4732-b581-4cfef3f01112"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:09:16.990845 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.990817    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-config-volume" (OuterVolumeSpecName: "config-volume") pod "08a3b90f-d6af-4732-b581-4cfef3f01112" (UID: "08a3b90f-d6af-4732-b581-4cfef3f01112"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:09:16.991200 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.991134    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "08a3b90f-d6af-4732-b581-4cfef3f01112" (UID: "08a3b90f-d6af-4732-b581-4cfef3f01112"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:09:16.991200 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.991167    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "08a3b90f-d6af-4732-b581-4cfef3f01112" (UID: "08a3b90f-d6af-4732-b581-4cfef3f01112"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:09:16.992275 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.992250    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "08a3b90f-d6af-4732-b581-4cfef3f01112" (UID: "08a3b90f-d6af-4732-b581-4cfef3f01112"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:09:16.999113 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:16.999086    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-web-config" (OuterVolumeSpecName: "web-config") pod "08a3b90f-d6af-4732-b581-4cfef3f01112" (UID: "08a3b90f-d6af-4732-b581-4cfef3f01112"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:09:17.084759 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.084664    2583 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-cluster-tls-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:17.084759 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.084706    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a3b90f-d6af-4732-b581-4cfef3f01112-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:17.084759 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.084718    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08a3b90f-d6af-4732-b581-4cfef3f01112-config-out\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:17.084759 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.084727    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh5nt\" (UniqueName: \"kubernetes.io/projected/08a3b90f-d6af-4732-b581-4cfef3f01112-kube-api-access-kh5nt\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:17.084759 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.084736    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08a3b90f-d6af-4732-b581-4cfef3f01112-tls-assets\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:17.084759 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.084744    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08a3b90f-d6af-4732-b581-4cfef3f01112-metrics-client-ca\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:17.084759 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.084752    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/08a3b90f-d6af-4732-b581-4cfef3f01112-alertmanager-main-db\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:17.084759 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.084760    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-main-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:17.084759 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.084770    2583 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-config-volume\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:17.085136 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.084778    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:17.085136 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.084786    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:17.085136 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.084794    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:17.085136 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.084803    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08a3b90f-d6af-4732-b581-4cfef3f01112-web-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:17.260998 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.260961    2583 generic.go:356] "Generic (PLEG): container finished" podID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerID="63b24679729fd33c93d3f4565fe718d9acdb3ff065d986658e33343e85a3012a" exitCode=0
Sep 17 19:09:17.261160 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.261033    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.261160 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.261043    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a3b90f-d6af-4732-b581-4cfef3f01112","Type":"ContainerDied","Data":"63b24679729fd33c93d3f4565fe718d9acdb3ff065d986658e33343e85a3012a"}
Sep 17 19:09:17.261160 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.261108    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a3b90f-d6af-4732-b581-4cfef3f01112","Type":"ContainerDied","Data":"5b77017fe61374bf85814270a2ce4f0c58e35c47baf3f7ca89ca5f80394c788d"}
Sep 17 19:09:17.261160 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.261139    2583 scope.go:117] "RemoveContainer" containerID="e4e98ff232ee7834cdb842e404f1577603a94ae69673ce38544a6f9612dc2566"
Sep 17 19:09:17.269030 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.269015    2583 scope.go:117] "RemoveContainer" containerID="ccbe3c3b0f813443dd871dea8759a1003bb4b5beae06c793a6bb90069df4ceda"
Sep 17 19:09:17.277223 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.277187    2583 scope.go:117] "RemoveContainer" containerID="73b3d7e3255a2a8213ce82ea6e4dcd4adea8bb2eedba0fb7ea3e366a0f44d73e"
Sep 17 19:09:17.285367 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.285349    2583 scope.go:117] "RemoveContainer" containerID="63b24679729fd33c93d3f4565fe718d9acdb3ff065d986658e33343e85a3012a"
Sep 17 19:09:17.290740 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.290718    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:09:17.293180 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.293160    2583 scope.go:117] "RemoveContainer" containerID="6e4a82bf11ade325771062efaa53acd00333d9a4390ea200cb03ec85b91247bb"
Sep 17 19:09:17.294375 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.294359    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:09:17.300042 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.300020    2583 scope.go:117] "RemoveContainer" containerID="676dd0a64537582a42e67827fef5f638f417dc4af1e4862d402b3bd31ff1eb0d"
Sep 17 19:09:17.306738 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.306723    2583 scope.go:117] "RemoveContainer" containerID="619359df720cefdebc69ea7d40ebf52853f21d85abe7db24dc2fc2bb17f32766"
Sep 17 19:09:17.313103 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.313089    2583 scope.go:117] "RemoveContainer" containerID="e4e98ff232ee7834cdb842e404f1577603a94ae69673ce38544a6f9612dc2566"
Sep 17 19:09:17.313350 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:09:17.313334    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e98ff232ee7834cdb842e404f1577603a94ae69673ce38544a6f9612dc2566\": container with ID starting with e4e98ff232ee7834cdb842e404f1577603a94ae69673ce38544a6f9612dc2566 not found: ID does not exist" containerID="e4e98ff232ee7834cdb842e404f1577603a94ae69673ce38544a6f9612dc2566"
Sep 17 19:09:17.313390 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.313356    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e98ff232ee7834cdb842e404f1577603a94ae69673ce38544a6f9612dc2566"} err="failed to get container status \"e4e98ff232ee7834cdb842e404f1577603a94ae69673ce38544a6f9612dc2566\": rpc error: code = NotFound desc = could not find container \"e4e98ff232ee7834cdb842e404f1577603a94ae69673ce38544a6f9612dc2566\": container with ID starting with e4e98ff232ee7834cdb842e404f1577603a94ae69673ce38544a6f9612dc2566 not found: ID does not exist"
Sep 17 19:09:17.313390 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.313374    2583 scope.go:117] "RemoveContainer" containerID="ccbe3c3b0f813443dd871dea8759a1003bb4b5beae06c793a6bb90069df4ceda"
Sep 17 19:09:17.313622 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:09:17.313605    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbe3c3b0f813443dd871dea8759a1003bb4b5beae06c793a6bb90069df4ceda\": container with ID starting with ccbe3c3b0f813443dd871dea8759a1003bb4b5beae06c793a6bb90069df4ceda not found: ID does not exist" containerID="ccbe3c3b0f813443dd871dea8759a1003bb4b5beae06c793a6bb90069df4ceda"
Sep 17 19:09:17.313665 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.313628    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbe3c3b0f813443dd871dea8759a1003bb4b5beae06c793a6bb90069df4ceda"} err="failed to get container status \"ccbe3c3b0f813443dd871dea8759a1003bb4b5beae06c793a6bb90069df4ceda\": rpc error: code = NotFound desc = could not find container \"ccbe3c3b0f813443dd871dea8759a1003bb4b5beae06c793a6bb90069df4ceda\": container with ID starting with ccbe3c3b0f813443dd871dea8759a1003bb4b5beae06c793a6bb90069df4ceda not found: ID does not exist"
Sep 17 19:09:17.313665 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.313640    2583 scope.go:117] "RemoveContainer" containerID="73b3d7e3255a2a8213ce82ea6e4dcd4adea8bb2eedba0fb7ea3e366a0f44d73e"
Sep 17 19:09:17.313886 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:09:17.313868    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b3d7e3255a2a8213ce82ea6e4dcd4adea8bb2eedba0fb7ea3e366a0f44d73e\": container with ID starting with 73b3d7e3255a2a8213ce82ea6e4dcd4adea8bb2eedba0fb7ea3e366a0f44d73e not found: ID does not exist" containerID="73b3d7e3255a2a8213ce82ea6e4dcd4adea8bb2eedba0fb7ea3e366a0f44d73e"
Sep 17 19:09:17.313922 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.313893    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b3d7e3255a2a8213ce82ea6e4dcd4adea8bb2eedba0fb7ea3e366a0f44d73e"} err="failed to get container status \"73b3d7e3255a2a8213ce82ea6e4dcd4adea8bb2eedba0fb7ea3e366a0f44d73e\": rpc error: code = NotFound desc = could not find container \"73b3d7e3255a2a8213ce82ea6e4dcd4adea8bb2eedba0fb7ea3e366a0f44d73e\": container with ID starting with 73b3d7e3255a2a8213ce82ea6e4dcd4adea8bb2eedba0fb7ea3e366a0f44d73e not found: ID does not exist"
Sep 17 19:09:17.313922 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.313908    2583 scope.go:117] "RemoveContainer" containerID="63b24679729fd33c93d3f4565fe718d9acdb3ff065d986658e33343e85a3012a"
Sep 17 19:09:17.314129 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:09:17.314115    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63b24679729fd33c93d3f4565fe718d9acdb3ff065d986658e33343e85a3012a\": container with ID starting with 63b24679729fd33c93d3f4565fe718d9acdb3ff065d986658e33343e85a3012a not found: ID does not exist" containerID="63b24679729fd33c93d3f4565fe718d9acdb3ff065d986658e33343e85a3012a"
Sep 17 19:09:17.314169 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.314132    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63b24679729fd33c93d3f4565fe718d9acdb3ff065d986658e33343e85a3012a"} err="failed to get container status \"63b24679729fd33c93d3f4565fe718d9acdb3ff065d986658e33343e85a3012a\": rpc error: code = NotFound desc = could not find container \"63b24679729fd33c93d3f4565fe718d9acdb3ff065d986658e33343e85a3012a\": container with ID starting with 63b24679729fd33c93d3f4565fe718d9acdb3ff065d986658e33343e85a3012a not found: ID does not exist"
Sep 17 19:09:17.314169 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.314148    2583 scope.go:117] "RemoveContainer" containerID="6e4a82bf11ade325771062efaa53acd00333d9a4390ea200cb03ec85b91247bb"
Sep 17 19:09:17.314389 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:09:17.314373    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e4a82bf11ade325771062efaa53acd00333d9a4390ea200cb03ec85b91247bb\": container with ID starting with 6e4a82bf11ade325771062efaa53acd00333d9a4390ea200cb03ec85b91247bb not found: ID does not exist" containerID="6e4a82bf11ade325771062efaa53acd00333d9a4390ea200cb03ec85b91247bb"
Sep 17 19:09:17.314426 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.314395    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e4a82bf11ade325771062efaa53acd00333d9a4390ea200cb03ec85b91247bb"} err="failed to get container status \"6e4a82bf11ade325771062efaa53acd00333d9a4390ea200cb03ec85b91247bb\": rpc error: code = NotFound desc = could not find container \"6e4a82bf11ade325771062efaa53acd00333d9a4390ea200cb03ec85b91247bb\": container with ID starting with 6e4a82bf11ade325771062efaa53acd00333d9a4390ea200cb03ec85b91247bb not found: ID does not exist"
Sep 17 19:09:17.314426 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.314412    2583 scope.go:117] "RemoveContainer" containerID="676dd0a64537582a42e67827fef5f638f417dc4af1e4862d402b3bd31ff1eb0d"
Sep 17 19:09:17.314659 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:09:17.314645    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676dd0a64537582a42e67827fef5f638f417dc4af1e4862d402b3bd31ff1eb0d\": container with ID starting with 676dd0a64537582a42e67827fef5f638f417dc4af1e4862d402b3bd31ff1eb0d not found: ID does not exist" containerID="676dd0a64537582a42e67827fef5f638f417dc4af1e4862d402b3bd31ff1eb0d"
Sep 17 19:09:17.314702 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.314663    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676dd0a64537582a42e67827fef5f638f417dc4af1e4862d402b3bd31ff1eb0d"} err="failed to get container status \"676dd0a64537582a42e67827fef5f638f417dc4af1e4862d402b3bd31ff1eb0d\": rpc error: code = NotFound desc = could not find container \"676dd0a64537582a42e67827fef5f638f417dc4af1e4862d402b3bd31ff1eb0d\": container with ID starting with 676dd0a64537582a42e67827fef5f638f417dc4af1e4862d402b3bd31ff1eb0d not found: ID does not exist"
Sep 17 19:09:17.314702 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.314675    2583 scope.go:117] "RemoveContainer" containerID="619359df720cefdebc69ea7d40ebf52853f21d85abe7db24dc2fc2bb17f32766"
Sep 17 19:09:17.314872 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:09:17.314852    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619359df720cefdebc69ea7d40ebf52853f21d85abe7db24dc2fc2bb17f32766\": container with ID starting with 619359df720cefdebc69ea7d40ebf52853f21d85abe7db24dc2fc2bb17f32766 not found: ID does not exist" containerID="619359df720cefdebc69ea7d40ebf52853f21d85abe7db24dc2fc2bb17f32766"
Sep 17 19:09:17.314909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.314876    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619359df720cefdebc69ea7d40ebf52853f21d85abe7db24dc2fc2bb17f32766"} err="failed to get container status \"619359df720cefdebc69ea7d40ebf52853f21d85abe7db24dc2fc2bb17f32766\": rpc error: code = NotFound desc = could not find container \"619359df720cefdebc69ea7d40ebf52853f21d85abe7db24dc2fc2bb17f32766\": container with ID starting with 619359df720cefdebc69ea7d40ebf52853f21d85abe7db24dc2fc2bb17f32766 not found: ID does not exist"
Sep 17 19:09:17.319723 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.319704    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:09:17.319984 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.319973    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="kube-rbac-proxy-web"
Sep 17 19:09:17.320022 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.319986    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="kube-rbac-proxy-web"
Sep 17 19:09:17.320022 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.319999    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="kube-rbac-proxy"
Sep 17 19:09:17.320022 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320005    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="kube-rbac-proxy"
Sep 17 19:09:17.320022 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320012    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="prom-label-proxy"
Sep 17 19:09:17.320022 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320017    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="prom-label-proxy"
Sep 17 19:09:17.320204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320026    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="config-reloader"
Sep 17 19:09:17.320204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320031    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="config-reloader"
Sep 17 19:09:17.320204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320037    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="init-config-reloader"
Sep 17 19:09:17.320204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320043    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="init-config-reloader"
Sep 17 19:09:17.320204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320049    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="kube-rbac-proxy-metric"
Sep 17 19:09:17.320204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320054    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="kube-rbac-proxy-metric"
Sep 17 19:09:17.320204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320064    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="alertmanager"
Sep 17 19:09:17.320204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320068    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="alertmanager"
Sep 17 19:09:17.320204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320107    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="alertmanager"
Sep 17 19:09:17.320204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320114    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="prom-label-proxy"
Sep 17 19:09:17.320204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320119    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="kube-rbac-proxy-web"
Sep 17 19:09:17.320204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320124    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="kube-rbac-proxy-metric"
Sep 17 19:09:17.320204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320129    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="config-reloader"
Sep 17 19:09:17.320204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.320135    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" containerName="kube-rbac-proxy"
Sep 17 19:09:17.323521 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.323506    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.326158 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.326142    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 19:09:17.326246 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.326141    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 19:09:17.326246 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.326144    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 19:09:17.326357 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.326144    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 19:09:17.326499 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.326485    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 19:09:17.326582 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.326567    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 19:09:17.326636 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.326618    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 19:09:17.326707 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.326694    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 19:09:17.327261 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.327225    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-z28dl\""
Sep 17 19:09:17.331189 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.331169    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 19:09:17.337374 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.337321    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:09:17.386868 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.386833    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.386868 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.386866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gfkg\" (UniqueName: \"kubernetes.io/projected/a2524f23-bfbf-4d32-bd23-50a9de492cba-kube-api-access-7gfkg\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.387081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.386894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-web-config\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.387081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.386914    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2524f23-bfbf-4d32-bd23-50a9de492cba-config-out\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.387081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.386932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.387081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.386951    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2524f23-bfbf-4d32-bd23-50a9de492cba-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.387081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.386988    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.387081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.387027    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.387081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.387057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.387284 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.387092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2524f23-bfbf-4d32-bd23-50a9de492cba-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.387284 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.387123    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2524f23-bfbf-4d32-bd23-50a9de492cba-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.387284 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.387138    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.387284 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.387167    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2524f23-bfbf-4d32-bd23-50a9de492cba-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.488173 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.488132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.488173 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.488175    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2524f23-bfbf-4d32-bd23-50a9de492cba-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.488414 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.488198    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2524f23-bfbf-4d32-bd23-50a9de492cba-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.488414 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.488219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.488414 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.488238    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2524f23-bfbf-4d32-bd23-50a9de492cba-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.488414 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.488271    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.488414 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.488290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gfkg\" (UniqueName: \"kubernetes.io/projected/a2524f23-bfbf-4d32-bd23-50a9de492cba-kube-api-access-7gfkg\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.488414 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.488326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-web-config\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.488746 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.488674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2524f23-bfbf-4d32-bd23-50a9de492cba-config-out\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.488746 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.488722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.488842 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.488752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2524f23-bfbf-4d32-bd23-50a9de492cba-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.488842 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.488786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.488842 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.488821    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.488991 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.488968    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2524f23-bfbf-4d32-bd23-50a9de492cba-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.489678 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.489377    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2524f23-bfbf-4d32-bd23-50a9de492cba-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.491329 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.491193    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2524f23-bfbf-4d32-bd23-50a9de492cba-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.491483 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.491360    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-web-config\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.491483 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.491415    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.491483 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.491473    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.491658 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.491570    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.491658 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.491605    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.491658 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.491604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.491810 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.491719    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2524f23-bfbf-4d32-bd23-50a9de492cba-config-out\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.492172 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.492150    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2524f23-bfbf-4d32-bd23-50a9de492cba-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.493148 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.493132    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2524f23-bfbf-4d32-bd23-50a9de492cba-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.496494 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.496479    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gfkg\" (UniqueName: \"kubernetes.io/projected/a2524f23-bfbf-4d32-bd23-50a9de492cba-kube-api-access-7gfkg\") pod \"alertmanager-main-0\" (UID: \"a2524f23-bfbf-4d32-bd23-50a9de492cba\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.633321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.633238    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 19:09:17.912068 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:17.912040    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 19:09:17.912394 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:09:17.912372    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2524f23_bfbf_4d32_bd23_50a9de492cba.slice/crio-c01e6c56cc751a2b53fddf7fa1c6288942cb718adf2e262c919a893015b1e7b3 WatchSource:0}: Error finding container c01e6c56cc751a2b53fddf7fa1c6288942cb718adf2e262c919a893015b1e7b3: Status 404 returned error can't find the container with id c01e6c56cc751a2b53fddf7fa1c6288942cb718adf2e262c919a893015b1e7b3
Sep 17 19:09:18.266660 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:18.266564    2583 generic.go:356] "Generic (PLEG): container finished" podID="a2524f23-bfbf-4d32-bd23-50a9de492cba" containerID="ab64f3cbc5feb45cc888e8734c0d73540fe5e9a7ed0468ef69ba068a760e84a2" exitCode=0
Sep 17 19:09:18.267054 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:18.266652    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2524f23-bfbf-4d32-bd23-50a9de492cba","Type":"ContainerDied","Data":"ab64f3cbc5feb45cc888e8734c0d73540fe5e9a7ed0468ef69ba068a760e84a2"}
Sep 17 19:09:18.267054 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:18.266740    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2524f23-bfbf-4d32-bd23-50a9de492cba","Type":"ContainerStarted","Data":"c01e6c56cc751a2b53fddf7fa1c6288942cb718adf2e262c919a893015b1e7b3"}
Sep 17 19:09:18.455225 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:18.455198    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08a3b90f-d6af-4732-b581-4cfef3f01112" path="/var/lib/kubelet/pods/08a3b90f-d6af-4732-b581-4cfef3f01112/volumes"
Sep 17 19:09:19.274806 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:19.274697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2524f23-bfbf-4d32-bd23-50a9de492cba","Type":"ContainerStarted","Data":"072afc90f2802720909cea9430e02d0e591809696ad1a580ab72bf1a6aff6f38"}
Sep 17 19:09:19.274806 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:19.274783    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2524f23-bfbf-4d32-bd23-50a9de492cba","Type":"ContainerStarted","Data":"1737bd1a2adaec95740b4dbd03261141c9cdd22a40f087ec3e34ea9694fb8eef"}
Sep 17 19:09:19.274806 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:19.274809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2524f23-bfbf-4d32-bd23-50a9de492cba","Type":"ContainerStarted","Data":"5e21b248a03937bc4de88c6a908462350599632d22ffb9cfd5cc19f648106c85"}
Sep 17 19:09:19.275193 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:19.274833    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2524f23-bfbf-4d32-bd23-50a9de492cba","Type":"ContainerStarted","Data":"cd85b092850f05a21b980a7ad997cbd160d7a33bc2cd1dc1b58bd20090d52a06"}
Sep 17 19:09:19.275193 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:19.274855    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2524f23-bfbf-4d32-bd23-50a9de492cba","Type":"ContainerStarted","Data":"bf170f3075933d72d33344535fc3eef28e5eb724a476b5c01dc32e2231b3c5e9"}
Sep 17 19:09:19.275193 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:19.274880    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2524f23-bfbf-4d32-bd23-50a9de492cba","Type":"ContainerStarted","Data":"a9f9fdf40004429a8dd8e96f2ac9354ad7d11611e22997f851e1b5f6f91ad1d0"}
Sep 17 19:09:19.306416 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:19.306371    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.291767079 podStartE2EDuration="2.306358059s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.59098ms" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:19.305091559 +0000 UTC m=+163.132949792" watchObservedRunningTime="2026-09-17 19:09:19.306358059 +0000 UTC m=+163.134216293"
Sep 17 19:09:21.765605 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:21.765570    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:21.765993 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:21.765630    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:21.770333 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:21.770311    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:22.290842 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:22.290817    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:09:22.350406 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:22.350374    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5797768c74-j2tqx"]
Sep 17 19:09:47.369155 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.369090    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5797768c74-j2tqx" podUID="2a1893f0-2284-4b8b-869d-37bf5c0b1f17" containerName="console" containerID="cri-o://156c6db44a416e98f4093714a64c57e476d270632337559ec08457880613cb33" gracePeriod=15
Sep 17 19:09:47.603548 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.603525    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:09:47.737623 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.737526    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-oauth-config\") pod \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") "
Sep 17 19:09:47.737623 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.737564    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-service-ca\") pod \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") "
Sep 17 19:09:47.737623 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.737580    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-serving-cert\") pod \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") "
Sep 17 19:09:47.737623 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.737618    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-oauth-serving-cert\") pod \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") "
Sep 17 19:09:47.737940 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.737640    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-trusted-ca-bundle\") pod \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") "
Sep 17 19:09:47.737940 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.737677    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxnk9\" (UniqueName: \"kubernetes.io/projected/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-kube-api-access-vxnk9\") pod \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") "
Sep 17 19:09:47.737940 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.737744    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-config\") pod \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\" (UID: \"2a1893f0-2284-4b8b-869d-37bf5c0b1f17\") "
Sep 17 19:09:47.738104 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.738043    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-service-ca" (OuterVolumeSpecName: "service-ca") pod "2a1893f0-2284-4b8b-869d-37bf5c0b1f17" (UID: "2a1893f0-2284-4b8b-869d-37bf5c0b1f17"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:09:47.738155 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.738128    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2a1893f0-2284-4b8b-869d-37bf5c0b1f17" (UID: "2a1893f0-2284-4b8b-869d-37bf5c0b1f17"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:09:47.738207 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.738167    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2a1893f0-2284-4b8b-869d-37bf5c0b1f17" (UID: "2a1893f0-2284-4b8b-869d-37bf5c0b1f17"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:09:47.738260 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.738203    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-config" (OuterVolumeSpecName: "console-config") pod "2a1893f0-2284-4b8b-869d-37bf5c0b1f17" (UID: "2a1893f0-2284-4b8b-869d-37bf5c0b1f17"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:09:47.739789 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.739761    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2a1893f0-2284-4b8b-869d-37bf5c0b1f17" (UID: "2a1893f0-2284-4b8b-869d-37bf5c0b1f17"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:09:47.740173 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.740144    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-kube-api-access-vxnk9" (OuterVolumeSpecName: "kube-api-access-vxnk9") pod "2a1893f0-2284-4b8b-869d-37bf5c0b1f17" (UID: "2a1893f0-2284-4b8b-869d-37bf5c0b1f17"). InnerVolumeSpecName "kube-api-access-vxnk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:09:47.740173 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.740146    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2a1893f0-2284-4b8b-869d-37bf5c0b1f17" (UID: "2a1893f0-2284-4b8b-869d-37bf5c0b1f17"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:09:47.838845 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.838813    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-oauth-serving-cert\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:47.838845 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.838840    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-trusted-ca-bundle\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:47.838845 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.838848    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxnk9\" (UniqueName: \"kubernetes.io/projected/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-kube-api-access-vxnk9\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:47.839050 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.838858    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:47.839050 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.838869    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-oauth-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:47.839050 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.838876    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-service-ca\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:47.839050 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:47.838884    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a1893f0-2284-4b8b-869d-37bf5c0b1f17-console-serving-cert\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:09:48.393605 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:48.393572    2583 generic.go:356] "Generic (PLEG): container finished" podID="2a1893f0-2284-4b8b-869d-37bf5c0b1f17" containerID="156c6db44a416e98f4093714a64c57e476d270632337559ec08457880613cb33" exitCode=0
Sep 17 19:09:48.393997 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:48.393621    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5797768c74-j2tqx"
Sep 17 19:09:48.393997 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:48.393658    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5797768c74-j2tqx" event={"ID":"2a1893f0-2284-4b8b-869d-37bf5c0b1f17","Type":"ContainerDied","Data":"156c6db44a416e98f4093714a64c57e476d270632337559ec08457880613cb33"}
Sep 17 19:09:48.393997 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:48.393711    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5797768c74-j2tqx" event={"ID":"2a1893f0-2284-4b8b-869d-37bf5c0b1f17","Type":"ContainerDied","Data":"0ad50390bcb1a034827c4d789b4acb26d3e26b8e0dd83aca74fff47333dc9308"}
Sep 17 19:09:48.393997 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:48.393738    2583 scope.go:117] "RemoveContainer" containerID="156c6db44a416e98f4093714a64c57e476d270632337559ec08457880613cb33"
Sep 17 19:09:48.401833 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:48.401811    2583 scope.go:117] "RemoveContainer" containerID="156c6db44a416e98f4093714a64c57e476d270632337559ec08457880613cb33"
Sep 17 19:09:48.402110 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:09:48.402089    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"156c6db44a416e98f4093714a64c57e476d270632337559ec08457880613cb33\": container with ID starting with 156c6db44a416e98f4093714a64c57e476d270632337559ec08457880613cb33 not found: ID does not exist" containerID="156c6db44a416e98f4093714a64c57e476d270632337559ec08457880613cb33"
Sep 17 19:09:48.402163 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:48.402119    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156c6db44a416e98f4093714a64c57e476d270632337559ec08457880613cb33"} err="failed to get container status \"156c6db44a416e98f4093714a64c57e476d270632337559ec08457880613cb33\": rpc error: code = NotFound desc = could not find container \"156c6db44a416e98f4093714a64c57e476d270632337559ec08457880613cb33\": container with ID starting with 156c6db44a416e98f4093714a64c57e476d270632337559ec08457880613cb33 not found: ID does not exist"
Sep 17 19:09:48.414822 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:48.414797    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5797768c74-j2tqx"]
Sep 17 19:09:48.419821 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:48.419802    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5797768c74-j2tqx"]
Sep 17 19:09:48.455014 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:09:48.454985    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a1893f0-2284-4b8b-869d-37bf5c0b1f17" path="/var/lib/kubelet/pods/2a1893f0-2284-4b8b-869d-37bf5c0b1f17/volumes"
Sep 17 19:10:19.653645 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.653615    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"]
Sep 17 19:10:19.654088 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.653891    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1893f0-2284-4b8b-869d-37bf5c0b1f17" containerName="console"
Sep 17 19:10:19.654088 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.653902    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1893f0-2284-4b8b-869d-37bf5c0b1f17" containerName="console"
Sep 17 19:10:19.654088 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.653954    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1893f0-2284-4b8b-869d-37bf5c0b1f17" containerName="console"
Sep 17 19:10:19.656187 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.656172    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"
Sep 17 19:10:19.658847 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.658828    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 19:10:19.658956 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.658832    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 19:10:19.658956 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.658864    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hr9mr\""
Sep 17 19:10:19.665951 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.665852    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"]
Sep 17 19:10:19.677483 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.677460    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b657057c-a2b8-4234-a771-ddb1a6ace4d0-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4\" (UID: \"b657057c-a2b8-4234-a771-ddb1a6ace4d0\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"
Sep 17 19:10:19.677592 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.677507    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b657057c-a2b8-4234-a771-ddb1a6ace4d0-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4\" (UID: \"b657057c-a2b8-4234-a771-ddb1a6ace4d0\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"
Sep 17 19:10:19.677592 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.677541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx9w7\" (UniqueName: \"kubernetes.io/projected/b657057c-a2b8-4234-a771-ddb1a6ace4d0-kube-api-access-mx9w7\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4\" (UID: \"b657057c-a2b8-4234-a771-ddb1a6ace4d0\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"
Sep 17 19:10:19.777929 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.777901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b657057c-a2b8-4234-a771-ddb1a6ace4d0-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4\" (UID: \"b657057c-a2b8-4234-a771-ddb1a6ace4d0\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"
Sep 17 19:10:19.777929 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.777934    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b657057c-a2b8-4234-a771-ddb1a6ace4d0-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4\" (UID: \"b657057c-a2b8-4234-a771-ddb1a6ace4d0\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"
Sep 17 19:10:19.778152 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.777951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9w7\" (UniqueName: \"kubernetes.io/projected/b657057c-a2b8-4234-a771-ddb1a6ace4d0-kube-api-access-mx9w7\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4\" (UID: \"b657057c-a2b8-4234-a771-ddb1a6ace4d0\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"
Sep 17 19:10:19.778386 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.778366    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b657057c-a2b8-4234-a771-ddb1a6ace4d0-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4\" (UID: \"b657057c-a2b8-4234-a771-ddb1a6ace4d0\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"
Sep 17 19:10:19.778424 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.778374    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b657057c-a2b8-4234-a771-ddb1a6ace4d0-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4\" (UID: \"b657057c-a2b8-4234-a771-ddb1a6ace4d0\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"
Sep 17 19:10:19.788624 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.788604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx9w7\" (UniqueName: \"kubernetes.io/projected/b657057c-a2b8-4234-a771-ddb1a6ace4d0-kube-api-access-mx9w7\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4\" (UID: \"b657057c-a2b8-4234-a771-ddb1a6ace4d0\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"
Sep 17 19:10:19.965317 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:19.965238    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"
Sep 17 19:10:20.232870 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:20.232792    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"]
Sep 17 19:10:20.236064 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:10:20.236039    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb657057c_a2b8_4234_a771_ddb1a6ace4d0.slice/crio-0f811395dd350dc52a934e215556a4d0dd66074025aaf5e8adb570588aa23a18 WatchSource:0}: Error finding container 0f811395dd350dc52a934e215556a4d0dd66074025aaf5e8adb570588aa23a18: Status 404 returned error can't find the container with id 0f811395dd350dc52a934e215556a4d0dd66074025aaf5e8adb570588aa23a18
Sep 17 19:10:20.510140 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:20.510029    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4" event={"ID":"b657057c-a2b8-4234-a771-ddb1a6ace4d0","Type":"ContainerStarted","Data":"0f811395dd350dc52a934e215556a4d0dd66074025aaf5e8adb570588aa23a18"}
Sep 17 19:10:26.534679 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:26.534648    2583 generic.go:356] "Generic (PLEG): container finished" podID="b657057c-a2b8-4234-a771-ddb1a6ace4d0" containerID="b34061a40edc8870bcad1138897751c649ef3a1c43c1011aa7ea909151840c16" exitCode=0
Sep 17 19:10:26.535047 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:26.534689    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4" event={"ID":"b657057c-a2b8-4234-a771-ddb1a6ace4d0","Type":"ContainerDied","Data":"b34061a40edc8870bcad1138897751c649ef3a1c43c1011aa7ea909151840c16"}
Sep 17 19:10:28.546070 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:28.546028    2583 generic.go:356] "Generic (PLEG): container finished" podID="b657057c-a2b8-4234-a771-ddb1a6ace4d0" containerID="aac7658e357a8f438539287d2f15ad01051a48e29de841a446ea58483e64d16e" exitCode=0
Sep 17 19:10:28.546526 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:28.546078    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4" event={"ID":"b657057c-a2b8-4234-a771-ddb1a6ace4d0","Type":"ContainerDied","Data":"aac7658e357a8f438539287d2f15ad01051a48e29de841a446ea58483e64d16e"}
Sep 17 19:10:35.576198 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:35.576163    2583 generic.go:356] "Generic (PLEG): container finished" podID="b657057c-a2b8-4234-a771-ddb1a6ace4d0" containerID="6bfd324de7f98c564ec9500bd8b79d38503b74e0352ad04925dc35e426477c64" exitCode=0
Sep 17 19:10:35.576198 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:35.576203    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4" event={"ID":"b657057c-a2b8-4234-a771-ddb1a6ace4d0","Type":"ContainerDied","Data":"6bfd324de7f98c564ec9500bd8b79d38503b74e0352ad04925dc35e426477c64"}
Sep 17 19:10:36.702683 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:36.702648    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"
Sep 17 19:10:36.823416 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:36.823379    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b657057c-a2b8-4234-a771-ddb1a6ace4d0-util\") pod \"b657057c-a2b8-4234-a771-ddb1a6ace4d0\" (UID: \"b657057c-a2b8-4234-a771-ddb1a6ace4d0\") "
Sep 17 19:10:36.823615 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:36.823431    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b657057c-a2b8-4234-a771-ddb1a6ace4d0-bundle\") pod \"b657057c-a2b8-4234-a771-ddb1a6ace4d0\" (UID: \"b657057c-a2b8-4234-a771-ddb1a6ace4d0\") "
Sep 17 19:10:36.823615 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:36.823488    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx9w7\" (UniqueName: \"kubernetes.io/projected/b657057c-a2b8-4234-a771-ddb1a6ace4d0-kube-api-access-mx9w7\") pod \"b657057c-a2b8-4234-a771-ddb1a6ace4d0\" (UID: \"b657057c-a2b8-4234-a771-ddb1a6ace4d0\") "
Sep 17 19:10:36.824191 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:36.824163    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b657057c-a2b8-4234-a771-ddb1a6ace4d0-bundle" (OuterVolumeSpecName: "bundle") pod "b657057c-a2b8-4234-a771-ddb1a6ace4d0" (UID: "b657057c-a2b8-4234-a771-ddb1a6ace4d0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:36.825630 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:36.825606    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b657057c-a2b8-4234-a771-ddb1a6ace4d0-kube-api-access-mx9w7" (OuterVolumeSpecName: "kube-api-access-mx9w7") pod "b657057c-a2b8-4234-a771-ddb1a6ace4d0" (UID: "b657057c-a2b8-4234-a771-ddb1a6ace4d0"). InnerVolumeSpecName "kube-api-access-mx9w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:10:36.827740 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:36.827719    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b657057c-a2b8-4234-a771-ddb1a6ace4d0-util" (OuterVolumeSpecName: "util") pod "b657057c-a2b8-4234-a771-ddb1a6ace4d0" (UID: "b657057c-a2b8-4234-a771-ddb1a6ace4d0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:36.924622 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:36.924537    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b657057c-a2b8-4234-a771-ddb1a6ace4d0-bundle\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:10:36.924622 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:36.924568    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx9w7\" (UniqueName: \"kubernetes.io/projected/b657057c-a2b8-4234-a771-ddb1a6ace4d0-kube-api-access-mx9w7\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:10:36.924622 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:36.924582    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b657057c-a2b8-4234-a771-ddb1a6ace4d0-util\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:10:37.586008 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:37.585981    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4"
Sep 17 19:10:37.586008 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:37.585982    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abzps4" event={"ID":"b657057c-a2b8-4234-a771-ddb1a6ace4d0","Type":"ContainerDied","Data":"0f811395dd350dc52a934e215556a4d0dd66074025aaf5e8adb570588aa23a18"}
Sep 17 19:10:37.586232 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:37.586029    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f811395dd350dc52a934e215556a4d0dd66074025aaf5e8adb570588aa23a18"
Sep 17 19:10:42.765744 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.765713    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69"]
Sep 17 19:10:42.766151 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.766011    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b657057c-a2b8-4234-a771-ddb1a6ace4d0" containerName="util"
Sep 17 19:10:42.766151 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.766021    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b657057c-a2b8-4234-a771-ddb1a6ace4d0" containerName="util"
Sep 17 19:10:42.766151 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.766032    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b657057c-a2b8-4234-a771-ddb1a6ace4d0" containerName="extract"
Sep 17 19:10:42.766151 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.766038    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b657057c-a2b8-4234-a771-ddb1a6ace4d0" containerName="extract"
Sep 17 19:10:42.766151 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.766055    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b657057c-a2b8-4234-a771-ddb1a6ace4d0" containerName="pull"
Sep 17 19:10:42.766151 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.766061    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b657057c-a2b8-4234-a771-ddb1a6ace4d0" containerName="pull"
Sep 17 19:10:42.766151 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.766144    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b657057c-a2b8-4234-a771-ddb1a6ace4d0" containerName="extract"
Sep 17 19:10:42.820135 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.820094    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69"]
Sep 17 19:10:42.820277 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.820215    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69"
Sep 17 19:10:42.826214 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.826185    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 19:10:42.826387 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.826212    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-2x47b\""
Sep 17 19:10:42.826387 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.826194    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 19:10:42.826387 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.826197    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 19:10:42.877651 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.877611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-227qh\" (UniqueName: \"kubernetes.io/projected/57efa572-c7d5-4dcb-b28e-0e5a353be93f-kube-api-access-227qh\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-5np69\" (UID: \"57efa572-c7d5-4dcb-b28e-0e5a353be93f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69"
Sep 17 19:10:42.877827 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.877660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/57efa572-c7d5-4dcb-b28e-0e5a353be93f-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-5np69\" (UID: \"57efa572-c7d5-4dcb-b28e-0e5a353be93f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69"
Sep 17 19:10:42.978432 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.978397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-227qh\" (UniqueName: \"kubernetes.io/projected/57efa572-c7d5-4dcb-b28e-0e5a353be93f-kube-api-access-227qh\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-5np69\" (UID: \"57efa572-c7d5-4dcb-b28e-0e5a353be93f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69"
Sep 17 19:10:42.978625 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.978470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/57efa572-c7d5-4dcb-b28e-0e5a353be93f-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-5np69\" (UID: \"57efa572-c7d5-4dcb-b28e-0e5a353be93f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69"
Sep 17 19:10:42.980803 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.980771    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/57efa572-c7d5-4dcb-b28e-0e5a353be93f-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-5np69\" (UID: \"57efa572-c7d5-4dcb-b28e-0e5a353be93f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69"
Sep 17 19:10:42.988731 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:42.988702    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-227qh\" (UniqueName: \"kubernetes.io/projected/57efa572-c7d5-4dcb-b28e-0e5a353be93f-kube-api-access-227qh\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-5np69\" (UID: \"57efa572-c7d5-4dcb-b28e-0e5a353be93f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69"
Sep 17 19:10:43.131083 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:43.131046    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69"
Sep 17 19:10:43.421848 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:43.421822    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69"]
Sep 17 19:10:43.423788 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:10:43.423763    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57efa572_c7d5_4dcb_b28e_0e5a353be93f.slice/crio-2fba8e2351af879c06c2090ec362a84c347fdb004069efdc2f0dc2f7e3395bc0 WatchSource:0}: Error finding container 2fba8e2351af879c06c2090ec362a84c347fdb004069efdc2f0dc2f7e3395bc0: Status 404 returned error can't find the container with id 2fba8e2351af879c06c2090ec362a84c347fdb004069efdc2f0dc2f7e3395bc0
Sep 17 19:10:43.610806 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:43.610773    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69" event={"ID":"57efa572-c7d5-4dcb-b28e-0e5a353be93f","Type":"ContainerStarted","Data":"2fba8e2351af879c06c2090ec362a84c347fdb004069efdc2f0dc2f7e3395bc0"}
Sep 17 19:10:48.635604 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:48.635562    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69" event={"ID":"57efa572-c7d5-4dcb-b28e-0e5a353be93f","Type":"ContainerStarted","Data":"f50cfb0e028a3070a4a9c4d10e7647aafee793b007c2dfdb924cef984dfc9ee4"}
Sep 17 19:10:48.636008 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:48.635659    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69"
Sep 17 19:10:48.659424 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:48.659375    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69" podStartSLOduration=1.673871074 podStartE2EDuration="6.659358622s" totalImagesPullingTime="4.985487548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:48.657134839 +0000 UTC m=+252.484993073" watchObservedRunningTime="2026-09-17 19:10:48.659358622 +0000 UTC m=+252.487216856"
Sep 17 19:10:49.260309 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:49.260229    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-rf6w4"]
Sep 17 19:10:49.275889 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:49.275854    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-rf6w4"
Sep 17 19:10:49.277968 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:49.277941    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-rf6w4"]
Sep 17 19:10:49.279002 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:49.278984    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-c4b9h\""
Sep 17 19:10:49.279659 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:49.279635    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 19:10:49.279758 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:49.279669    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 19:10:49.431146 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:49.431101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkd2k\" (UniqueName: \"kubernetes.io/projected/9fbde9f3-abc5-4c1a-971a-5535f169ac05-kube-api-access-mkd2k\") pod \"keda-operator-579c8958f7-rf6w4\" (UID: \"9fbde9f3-abc5-4c1a-971a-5535f169ac05\") " pod="openshift-keda/keda-operator-579c8958f7-rf6w4"
Sep 17 19:10:49.431311 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:49.431223    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9fbde9f3-abc5-4c1a-971a-5535f169ac05-cabundle0\") pod \"keda-operator-579c8958f7-rf6w4\" (UID: \"9fbde9f3-abc5-4c1a-971a-5535f169ac05\") " pod="openshift-keda/keda-operator-579c8958f7-rf6w4"
Sep 17 19:10:49.431311 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:49.431260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9fbde9f3-abc5-4c1a-971a-5535f169ac05-certificates\") pod \"keda-operator-579c8958f7-rf6w4\" (UID: \"9fbde9f3-abc5-4c1a-971a-5535f169ac05\") " pod="openshift-keda/keda-operator-579c8958f7-rf6w4"
Sep 17 19:10:49.531886 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:49.531803    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9fbde9f3-abc5-4c1a-971a-5535f169ac05-cabundle0\") pod \"keda-operator-579c8958f7-rf6w4\" (UID: \"9fbde9f3-abc5-4c1a-971a-5535f169ac05\") " pod="openshift-keda/keda-operator-579c8958f7-rf6w4"
Sep 17 19:10:49.531886 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:49.531840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9fbde9f3-abc5-4c1a-971a-5535f169ac05-certificates\") pod \"keda-operator-579c8958f7-rf6w4\" (UID: \"9fbde9f3-abc5-4c1a-971a-5535f169ac05\") " pod="openshift-keda/keda-operator-579c8958f7-rf6w4"
Sep 17 19:10:49.531886 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:49.531879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkd2k\" (UniqueName: \"kubernetes.io/projected/9fbde9f3-abc5-4c1a-971a-5535f169ac05-kube-api-access-mkd2k\") pod \"keda-operator-579c8958f7-rf6w4\" (UID: \"9fbde9f3-abc5-4c1a-971a-5535f169ac05\") " pod="openshift-keda/keda-operator-579c8958f7-rf6w4"
Sep 17 19:10:49.532192 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:10:49.531974    2583 secret.go:281] references non-existent secret key: ca.crt
Sep 17 19:10:49.532192 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:10:49.531992    2583 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 19:10:49.532192 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:10:49.532002    2583 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-rf6w4: references non-existent secret key: ca.crt
Sep 17 19:10:49.532192 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:10:49.532059    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9fbde9f3-abc5-4c1a-971a-5535f169ac05-certificates podName:9fbde9f3-abc5-4c1a-971a-5535f169ac05 nodeName:}" failed. No retries permitted until 2026-09-17 19:10:50.03204217 +0000 UTC m=+253.859900396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/9fbde9f3-abc5-4c1a-971a-5535f169ac05-certificates") pod "keda-operator-579c8958f7-rf6w4" (UID: "9fbde9f3-abc5-4c1a-971a-5535f169ac05") : references non-existent secret key: ca.crt
Sep 17 19:10:49.532578 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:49.532554    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9fbde9f3-abc5-4c1a-971a-5535f169ac05-cabundle0\") pod \"keda-operator-579c8958f7-rf6w4\" (UID: \"9fbde9f3-abc5-4c1a-971a-5535f169ac05\") " pod="openshift-keda/keda-operator-579c8958f7-rf6w4"
Sep 17 19:10:49.546984 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:49.546950    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkd2k\" (UniqueName: \"kubernetes.io/projected/9fbde9f3-abc5-4c1a-971a-5535f169ac05-kube-api-access-mkd2k\") pod \"keda-operator-579c8958f7-rf6w4\" (UID: \"9fbde9f3-abc5-4c1a-971a-5535f169ac05\") " pod="openshift-keda/keda-operator-579c8958f7-rf6w4"
Sep 17 19:10:50.016516 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.016480    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"]
Sep 17 19:10:50.018720 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.018704    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"
Sep 17 19:10:50.021771 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.021749    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 19:10:50.035400 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.035370    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"]
Sep 17 19:10:50.035788 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.035757    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9fbde9f3-abc5-4c1a-971a-5535f169ac05-certificates\") pod \"keda-operator-579c8958f7-rf6w4\" (UID: \"9fbde9f3-abc5-4c1a-971a-5535f169ac05\") " pod="openshift-keda/keda-operator-579c8958f7-rf6w4"
Sep 17 19:10:50.038387 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.038361    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9fbde9f3-abc5-4c1a-971a-5535f169ac05-certificates\") pod \"keda-operator-579c8958f7-rf6w4\" (UID: \"9fbde9f3-abc5-4c1a-971a-5535f169ac05\") " pod="openshift-keda/keda-operator-579c8958f7-rf6w4"
Sep 17 19:10:50.137255 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.137216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2b09925b-1398-4129-9612-30f3848f8bf5-certificates\") pod \"keda-metrics-apiserver-77857b7b98-dbmr6\" (UID: \"2b09925b-1398-4129-9612-30f3848f8bf5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"
Sep 17 19:10:50.137255 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.137263    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/2b09925b-1398-4129-9612-30f3848f8bf5-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-dbmr6\" (UID: \"2b09925b-1398-4129-9612-30f3848f8bf5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"
Sep 17 19:10:50.137516 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.137326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62v9z\" (UniqueName: \"kubernetes.io/projected/2b09925b-1398-4129-9612-30f3848f8bf5-kube-api-access-62v9z\") pod \"keda-metrics-apiserver-77857b7b98-dbmr6\" (UID: \"2b09925b-1398-4129-9612-30f3848f8bf5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"
Sep 17 19:10:50.187538 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.187508    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-rf6w4"
Sep 17 19:10:50.239215 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.239018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2b09925b-1398-4129-9612-30f3848f8bf5-certificates\") pod \"keda-metrics-apiserver-77857b7b98-dbmr6\" (UID: \"2b09925b-1398-4129-9612-30f3848f8bf5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"
Sep 17 19:10:50.239215 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.239070    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/2b09925b-1398-4129-9612-30f3848f8bf5-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-dbmr6\" (UID: \"2b09925b-1398-4129-9612-30f3848f8bf5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"
Sep 17 19:10:50.239215 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.239123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62v9z\" (UniqueName: \"kubernetes.io/projected/2b09925b-1398-4129-9612-30f3848f8bf5-kube-api-access-62v9z\") pod \"keda-metrics-apiserver-77857b7b98-dbmr6\" (UID: \"2b09925b-1398-4129-9612-30f3848f8bf5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"
Sep 17 19:10:50.240116 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.239830    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/2b09925b-1398-4129-9612-30f3848f8bf5-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-dbmr6\" (UID: \"2b09925b-1398-4129-9612-30f3848f8bf5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"
Sep 17 19:10:50.242635 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.242615    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2b09925b-1398-4129-9612-30f3848f8bf5-certificates\") pod \"keda-metrics-apiserver-77857b7b98-dbmr6\" (UID: \"2b09925b-1398-4129-9612-30f3848f8bf5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"
Sep 17 19:10:50.290982 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.290948    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62v9z\" (UniqueName: \"kubernetes.io/projected/2b09925b-1398-4129-9612-30f3848f8bf5-kube-api-access-62v9z\") pod \"keda-metrics-apiserver-77857b7b98-dbmr6\" (UID: \"2b09925b-1398-4129-9612-30f3848f8bf5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"
Sep 17 19:10:50.328927 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.328897    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"
Sep 17 19:10:50.494626 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.494569    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-rf6w4"]
Sep 17 19:10:50.497321 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:10:50.497296    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbde9f3_abc5_4c1a_971a_5535f169ac05.slice/crio-ca18db66a624eba5390b118b060a1f4e1c8197f3bd43c7d6e07651ddba596496 WatchSource:0}: Error finding container ca18db66a624eba5390b118b060a1f4e1c8197f3bd43c7d6e07651ddba596496: Status 404 returned error can't find the container with id ca18db66a624eba5390b118b060a1f4e1c8197f3bd43c7d6e07651ddba596496
Sep 17 19:10:50.638556 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.638478    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"]
Sep 17 19:10:50.641809 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:10:50.641777    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b09925b_1398_4129_9612_30f3848f8bf5.slice/crio-dabe1458c3665634090006adc8c9ed39d6b86c01559fe9743f5aa3a57ca93d15 WatchSource:0}: Error finding container dabe1458c3665634090006adc8c9ed39d6b86c01559fe9743f5aa3a57ca93d15: Status 404 returned error can't find the container with id dabe1458c3665634090006adc8c9ed39d6b86c01559fe9743f5aa3a57ca93d15
Sep 17 19:10:50.646895 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:50.646866    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-rf6w4" event={"ID":"9fbde9f3-abc5-4c1a-971a-5535f169ac05","Type":"ContainerStarted","Data":"ca18db66a624eba5390b118b060a1f4e1c8197f3bd43c7d6e07651ddba596496"}
Sep 17 19:10:51.652185 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:51.652153    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6" event={"ID":"2b09925b-1398-4129-9612-30f3848f8bf5","Type":"ContainerStarted","Data":"dabe1458c3665634090006adc8c9ed39d6b86c01559fe9743f5aa3a57ca93d15"}
Sep 17 19:10:56.677752 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:56.677720    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-rf6w4" event={"ID":"9fbde9f3-abc5-4c1a-971a-5535f169ac05","Type":"ContainerStarted","Data":"f8c8a70c313b7e229c5c8fcbfd8e09da430ccbec002470ad8627ce06c9d69d25"}
Sep 17 19:10:56.678215 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:56.677869    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-rf6w4"
Sep 17 19:10:56.679435 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:56.679410    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6" event={"ID":"2b09925b-1398-4129-9612-30f3848f8bf5","Type":"ContainerStarted","Data":"e629d7b4227f94cce29c6d526f901e40d6738416c31b79a50493a9829af7a625"}
Sep 17 19:10:56.679586 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:56.679485    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"
Sep 17 19:10:56.701355 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:56.701307    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-rf6w4" podStartSLOduration=2.149981765 podStartE2EDuration="7.701289448s" totalImagesPullingTime="5.551307683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:56.700762834 +0000 UTC m=+260.528621080" watchObservedRunningTime="2026-09-17 19:10:56.701289448 +0000 UTC m=+260.529147683"
Sep 17 19:10:56.725060 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:10:56.725006    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6" podStartSLOduration=2.32274152 podStartE2EDuration="7.724988729s" totalImagesPullingTime="5.402247209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:56.723416047 +0000 UTC m=+260.551274283" watchObservedRunningTime="2026-09-17 19:10:56.724988729 +0000 UTC m=+260.552846964"
Sep 17 19:11:07.690687 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:11:07.690648    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dbmr6"
Sep 17 19:11:09.645081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:11:09.645050    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-5np69"
Sep 17 19:11:17.687128 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:11:17.687096    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-rf6w4"
Sep 17 19:11:38.309861 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:11:38.309829    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:11:38.311399 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:11:38.311377    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:11:38.312855 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:11:38.312840    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 19:12:06.646095 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.646057    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf"]
Sep 17 19:12:06.648332 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.648314    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf"
Sep 17 19:12:06.653769 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.653746    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 19:12:06.654475 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.654040    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 19:12:06.654643 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.654622    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 19:12:06.654761 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.654692    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-6pdnb\""
Sep 17 19:12:06.670780 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.670746    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf"]
Sep 17 19:12:06.714140 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.714108    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-cnljh"]
Sep 17 19:12:06.716725 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.716702    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-cnljh"
Sep 17 19:12:06.719575 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.719551    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-nhdv6\""
Sep 17 19:12:06.719722 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.719651    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 19:12:06.736184 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.736148    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-cnljh"]
Sep 17 19:12:06.784547 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.784504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91f2b8d6-c5d4-4288-ae95-3a68d992e56a-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-pwhcf\" (UID: \"91f2b8d6-c5d4-4288-ae95-3a68d992e56a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf"
Sep 17 19:12:06.784727 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.784628    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p52j9\" (UniqueName: \"kubernetes.io/projected/91f2b8d6-c5d4-4288-ae95-3a68d992e56a-kube-api-access-p52j9\") pod \"llmisvc-controller-manager-645f9c8fd6-pwhcf\" (UID: \"91f2b8d6-c5d4-4288-ae95-3a68d992e56a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf"
Sep 17 19:12:06.885680 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.885636    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qztcz\" (UniqueName: \"kubernetes.io/projected/121beb01-4aac-48ee-b683-59bc4acc439b-kube-api-access-qztcz\") pod \"seaweedfs-64568bcd49-cnljh\" (UID: \"121beb01-4aac-48ee-b683-59bc4acc439b\") " pod="kserve/seaweedfs-64568bcd49-cnljh"
Sep 17 19:12:06.885680 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.885677    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/121beb01-4aac-48ee-b683-59bc4acc439b-data\") pod \"seaweedfs-64568bcd49-cnljh\" (UID: \"121beb01-4aac-48ee-b683-59bc4acc439b\") " pod="kserve/seaweedfs-64568bcd49-cnljh"
Sep 17 19:12:06.885889 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.885706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p52j9\" (UniqueName: \"kubernetes.io/projected/91f2b8d6-c5d4-4288-ae95-3a68d992e56a-kube-api-access-p52j9\") pod \"llmisvc-controller-manager-645f9c8fd6-pwhcf\" (UID: \"91f2b8d6-c5d4-4288-ae95-3a68d992e56a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf"
Sep 17 19:12:06.885889 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.885801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91f2b8d6-c5d4-4288-ae95-3a68d992e56a-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-pwhcf\" (UID: \"91f2b8d6-c5d4-4288-ae95-3a68d992e56a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf"
Sep 17 19:12:06.888301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.888275    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91f2b8d6-c5d4-4288-ae95-3a68d992e56a-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-pwhcf\" (UID: \"91f2b8d6-c5d4-4288-ae95-3a68d992e56a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf"
Sep 17 19:12:06.895540 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.895509    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p52j9\" (UniqueName: \"kubernetes.io/projected/91f2b8d6-c5d4-4288-ae95-3a68d992e56a-kube-api-access-p52j9\") pod \"llmisvc-controller-manager-645f9c8fd6-pwhcf\" (UID: \"91f2b8d6-c5d4-4288-ae95-3a68d992e56a\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf"
Sep 17 19:12:06.958101 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.958002    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf"
Sep 17 19:12:06.986552 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.986523    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qztcz\" (UniqueName: \"kubernetes.io/projected/121beb01-4aac-48ee-b683-59bc4acc439b-kube-api-access-qztcz\") pod \"seaweedfs-64568bcd49-cnljh\" (UID: \"121beb01-4aac-48ee-b683-59bc4acc439b\") " pod="kserve/seaweedfs-64568bcd49-cnljh"
Sep 17 19:12:06.986552 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.986554    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/121beb01-4aac-48ee-b683-59bc4acc439b-data\") pod \"seaweedfs-64568bcd49-cnljh\" (UID: \"121beb01-4aac-48ee-b683-59bc4acc439b\") " pod="kserve/seaweedfs-64568bcd49-cnljh"
Sep 17 19:12:06.986919 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.986900    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/121beb01-4aac-48ee-b683-59bc4acc439b-data\") pod \"seaweedfs-64568bcd49-cnljh\" (UID: \"121beb01-4aac-48ee-b683-59bc4acc439b\") " pod="kserve/seaweedfs-64568bcd49-cnljh"
Sep 17 19:12:06.996072 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:06.996015    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qztcz\" (UniqueName: \"kubernetes.io/projected/121beb01-4aac-48ee-b683-59bc4acc439b-kube-api-access-qztcz\") pod \"seaweedfs-64568bcd49-cnljh\" (UID: \"121beb01-4aac-48ee-b683-59bc4acc439b\") " pod="kserve/seaweedfs-64568bcd49-cnljh"
Sep 17 19:12:07.025652 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:07.025620    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-cnljh"
Sep 17 19:12:07.230455 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:07.230410    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf"]
Sep 17 19:12:07.232005 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:12:07.231979    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod91f2b8d6_c5d4_4288_ae95_3a68d992e56a.slice/crio-dc7cf3f26b43be5739e86d64991bef47bcc39b0c897da94d107e23df1e8ca956 WatchSource:0}: Error finding container dc7cf3f26b43be5739e86d64991bef47bcc39b0c897da94d107e23df1e8ca956: Status 404 returned error can't find the container with id dc7cf3f26b43be5739e86d64991bef47bcc39b0c897da94d107e23df1e8ca956
Sep 17 19:12:07.233322 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:07.233303    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:12:07.298418 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:07.298265    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-cnljh"]
Sep 17 19:12:07.300961 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:12:07.300929    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121beb01_4aac_48ee_b683_59bc4acc439b.slice/crio-a0ecb410b2681040b8754457909c274451582b3a4f00cee0864eff5f42111e1f WatchSource:0}: Error finding container a0ecb410b2681040b8754457909c274451582b3a4f00cee0864eff5f42111e1f: Status 404 returned error can't find the container with id a0ecb410b2681040b8754457909c274451582b3a4f00cee0864eff5f42111e1f
Sep 17 19:12:07.996022 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:07.995981    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf" event={"ID":"91f2b8d6-c5d4-4288-ae95-3a68d992e56a","Type":"ContainerStarted","Data":"dc7cf3f26b43be5739e86d64991bef47bcc39b0c897da94d107e23df1e8ca956"}
Sep 17 19:12:07.998053 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:07.997976    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-cnljh" event={"ID":"121beb01-4aac-48ee-b683-59bc4acc439b","Type":"ContainerStarted","Data":"a0ecb410b2681040b8754457909c274451582b3a4f00cee0864eff5f42111e1f"}
Sep 17 19:12:11.015668 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:11.015625    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf" event={"ID":"91f2b8d6-c5d4-4288-ae95-3a68d992e56a","Type":"ContainerStarted","Data":"f1f6d58081295c2c10ce724bd91c0c2bf98b3839ce903e49ecbe16281348a1b8"}
Sep 17 19:12:11.016181 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:11.015720    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf"
Sep 17 19:12:11.017268 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:11.017245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-cnljh" event={"ID":"121beb01-4aac-48ee-b683-59bc4acc439b","Type":"ContainerStarted","Data":"cf61bc07172afebc023a9d12f891a38510972ecd3f642906415486c940b1295c"}
Sep 17 19:12:11.017388 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:11.017295    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-cnljh"
Sep 17 19:12:11.034264 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:11.034213    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf" podStartSLOduration=1.635057941 podStartE2EDuration="5.03419753s" totalImagesPullingTime="3.399139589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:12:11.032391119 +0000 UTC m=+334.860249354" watchObservedRunningTime="2026-09-17 19:12:11.03419753 +0000 UTC m=+334.862055764"
Sep 17 19:12:11.049912 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:11.049852    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-cnljh" podStartSLOduration=1.50923944 podStartE2EDuration="5.049835656s" totalImagesPullingTime="3.540596216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:12:11.048641548 +0000 UTC m=+334.876499788" watchObservedRunningTime="2026-09-17 19:12:11.049835656 +0000 UTC m=+334.877693891"
Sep 17 19:12:17.024310 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:17.024280    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-cnljh"
Sep 17 19:12:42.024558 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:12:42.024483    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-pwhcf"
Sep 17 19:13:00.722772 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.722736    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d7fddb4ff-r9nhq"]
Sep 17 19:13:00.725211 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.725188    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.741304 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.741276    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d7fddb4ff-r9nhq"]
Sep 17 19:13:00.830453 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.830415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsrgw\" (UniqueName: \"kubernetes.io/projected/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-kube-api-access-gsrgw\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.830620 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.830482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-console-config\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.830620 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.830531    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-console-oauth-config\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.830620 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.830571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-service-ca\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.830620 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.830585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-oauth-serving-cert\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.830752 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.830633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-trusted-ca-bundle\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.830752 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.830660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-console-serving-cert\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.931778 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.931739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-console-serving-cert\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.931778 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.931774    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsrgw\" (UniqueName: \"kubernetes.io/projected/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-kube-api-access-gsrgw\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.932019 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.931793    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-console-config\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.932019 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.931927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-console-oauth-config\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.932019 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.931980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-service-ca\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.932019 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.932003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-oauth-serving-cert\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.932225 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.932070    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-trusted-ca-bundle\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.932645 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.932618    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-console-config\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.932787 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.932708    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-service-ca\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.932935 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.932913    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-oauth-serving-cert\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.933000 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.932913    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-trusted-ca-bundle\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.934405 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.934381    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-console-oauth-config\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.934510 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.934496    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-console-serving-cert\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:00.941539 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:00.941520    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsrgw\" (UniqueName: \"kubernetes.io/projected/c3b00832-730b-498a-a79a-fa5b6ca6d7eb-kube-api-access-gsrgw\") pod \"console-7d7fddb4ff-r9nhq\" (UID: \"c3b00832-730b-498a-a79a-fa5b6ca6d7eb\") " pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:01.034769 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:01.034738    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:01.304577 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:01.304554    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d7fddb4ff-r9nhq"]
Sep 17 19:13:01.307340 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:13:01.307312    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b00832_730b_498a_a79a_fa5b6ca6d7eb.slice/crio-9931908a67ecdf1d67c20fd8d3f1456d58e40ded36963735d948835aee2e0a28 WatchSource:0}: Error finding container 9931908a67ecdf1d67c20fd8d3f1456d58e40ded36963735d948835aee2e0a28: Status 404 returned error can't find the container with id 9931908a67ecdf1d67c20fd8d3f1456d58e40ded36963735d948835aee2e0a28
Sep 17 19:13:02.236976 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:02.236941    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d7fddb4ff-r9nhq" event={"ID":"c3b00832-730b-498a-a79a-fa5b6ca6d7eb","Type":"ContainerStarted","Data":"27bf533058a016621db254526061331a7b38b5b8112fc4fc1686c9ff6e115ef2"}
Sep 17 19:13:02.237489 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:02.237034    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d7fddb4ff-r9nhq" event={"ID":"c3b00832-730b-498a-a79a-fa5b6ca6d7eb","Type":"ContainerStarted","Data":"9931908a67ecdf1d67c20fd8d3f1456d58e40ded36963735d948835aee2e0a28"}
Sep 17 19:13:02.257327 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:02.257273    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d7fddb4ff-r9nhq" podStartSLOduration=2.257256456 podStartE2EDuration="2.257256456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:02.25517465 +0000 UTC m=+386.083032885" watchObservedRunningTime="2026-09-17 19:13:02.257256456 +0000 UTC m=+386.085114690"
Sep 17 19:13:11.035024 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:11.034983    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:11.035504 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:11.035270    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:11.040066 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:11.040043    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:11.278080 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:11.278054    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d7fddb4ff-r9nhq"
Sep 17 19:13:11.324383 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:11.324308    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c889d49bb-fxp5r"]
Sep 17 19:13:36.348000 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.347935    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c889d49bb-fxp5r" podUID="9f08e542-7c5f-459d-970f-8cc3a34b3546" containerName="console" containerID="cri-o://76eb57e2c475fc35cc22e728269f6ce6e518b5ec7131ffe3ab6a57c147f9635d" gracePeriod=15
Sep 17 19:13:36.593273 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.593249    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:13:36.731698 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.731613    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-serving-cert\") pod \"9f08e542-7c5f-459d-970f-8cc3a34b3546\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") "
Sep 17 19:13:36.731698 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.731678    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-oauth-serving-cert\") pod \"9f08e542-7c5f-459d-970f-8cc3a34b3546\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") "
Sep 17 19:13:36.731698 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.731693    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-oauth-config\") pod \"9f08e542-7c5f-459d-970f-8cc3a34b3546\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") "
Sep 17 19:13:36.731961 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.731712    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-trusted-ca-bundle\") pod \"9f08e542-7c5f-459d-970f-8cc3a34b3546\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") "
Sep 17 19:13:36.731961 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.731873    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-config\") pod \"9f08e542-7c5f-459d-970f-8cc3a34b3546\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") "
Sep 17 19:13:36.731961 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.731916    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-service-ca\") pod \"9f08e542-7c5f-459d-970f-8cc3a34b3546\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") "
Sep 17 19:13:36.732115 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.732034    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9f08e542-7c5f-459d-970f-8cc3a34b3546" (UID: "9f08e542-7c5f-459d-970f-8cc3a34b3546"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:36.732115 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.732056    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m78lk\" (UniqueName: \"kubernetes.io/projected/9f08e542-7c5f-459d-970f-8cc3a34b3546-kube-api-access-m78lk\") pod \"9f08e542-7c5f-459d-970f-8cc3a34b3546\" (UID: \"9f08e542-7c5f-459d-970f-8cc3a34b3546\") "
Sep 17 19:13:36.732247 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.732137    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9f08e542-7c5f-459d-970f-8cc3a34b3546" (UID: "9f08e542-7c5f-459d-970f-8cc3a34b3546"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:36.732328 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.732275    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-config" (OuterVolumeSpecName: "console-config") pod "9f08e542-7c5f-459d-970f-8cc3a34b3546" (UID: "9f08e542-7c5f-459d-970f-8cc3a34b3546"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:36.732328 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.732305    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-oauth-serving-cert\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:13:36.732328 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.732303    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-service-ca" (OuterVolumeSpecName: "service-ca") pod "9f08e542-7c5f-459d-970f-8cc3a34b3546" (UID: "9f08e542-7c5f-459d-970f-8cc3a34b3546"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:36.732328 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.732324    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-trusted-ca-bundle\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:13:36.733917 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.733886    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9f08e542-7c5f-459d-970f-8cc3a34b3546" (UID: "9f08e542-7c5f-459d-970f-8cc3a34b3546"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:13:36.734028 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.733927    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9f08e542-7c5f-459d-970f-8cc3a34b3546" (UID: "9f08e542-7c5f-459d-970f-8cc3a34b3546"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:13:36.734083 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.734043    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f08e542-7c5f-459d-970f-8cc3a34b3546-kube-api-access-m78lk" (OuterVolumeSpecName: "kube-api-access-m78lk") pod "9f08e542-7c5f-459d-970f-8cc3a34b3546" (UID: "9f08e542-7c5f-459d-970f-8cc3a34b3546"). InnerVolumeSpecName "kube-api-access-m78lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:36.833655 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.833623    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m78lk\" (UniqueName: \"kubernetes.io/projected/9f08e542-7c5f-459d-970f-8cc3a34b3546-kube-api-access-m78lk\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:13:36.833655 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.833649    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-serving-cert\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:13:36.833655 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.833659    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-oauth-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:13:36.833655 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.833668    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-console-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:13:36.833909 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:36.833676    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f08e542-7c5f-459d-970f-8cc3a34b3546-service-ca\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:13:37.386222 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:37.386186    2583 generic.go:356] "Generic (PLEG): container finished" podID="9f08e542-7c5f-459d-970f-8cc3a34b3546" containerID="76eb57e2c475fc35cc22e728269f6ce6e518b5ec7131ffe3ab6a57c147f9635d" exitCode=0
Sep 17 19:13:37.386693 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:37.386234    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c889d49bb-fxp5r"
Sep 17 19:13:37.386693 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:37.386274    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c889d49bb-fxp5r" event={"ID":"9f08e542-7c5f-459d-970f-8cc3a34b3546","Type":"ContainerDied","Data":"76eb57e2c475fc35cc22e728269f6ce6e518b5ec7131ffe3ab6a57c147f9635d"}
Sep 17 19:13:37.386693 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:37.386361    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c889d49bb-fxp5r" event={"ID":"9f08e542-7c5f-459d-970f-8cc3a34b3546","Type":"ContainerDied","Data":"0f215529cfe079c3622a9f3c79d28c16110edd1e1e2c084367e45cc619a09478"}
Sep 17 19:13:37.386693 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:37.386392    2583 scope.go:117] "RemoveContainer" containerID="76eb57e2c475fc35cc22e728269f6ce6e518b5ec7131ffe3ab6a57c147f9635d"
Sep 17 19:13:37.394812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:37.394797    2583 scope.go:117] "RemoveContainer" containerID="76eb57e2c475fc35cc22e728269f6ce6e518b5ec7131ffe3ab6a57c147f9635d"
Sep 17 19:13:37.395055 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:13:37.395039    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76eb57e2c475fc35cc22e728269f6ce6e518b5ec7131ffe3ab6a57c147f9635d\": container with ID starting with 76eb57e2c475fc35cc22e728269f6ce6e518b5ec7131ffe3ab6a57c147f9635d not found: ID does not exist" containerID="76eb57e2c475fc35cc22e728269f6ce6e518b5ec7131ffe3ab6a57c147f9635d"
Sep 17 19:13:37.395103 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:37.395062    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76eb57e2c475fc35cc22e728269f6ce6e518b5ec7131ffe3ab6a57c147f9635d"} err="failed to get container status \"76eb57e2c475fc35cc22e728269f6ce6e518b5ec7131ffe3ab6a57c147f9635d\": rpc error: code = NotFound desc = could not find container \"76eb57e2c475fc35cc22e728269f6ce6e518b5ec7131ffe3ab6a57c147f9635d\": container with ID starting with 76eb57e2c475fc35cc22e728269f6ce6e518b5ec7131ffe3ab6a57c147f9635d not found: ID does not exist"
Sep 17 19:13:37.407017 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:37.406997    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c889d49bb-fxp5r"]
Sep 17 19:13:37.409154 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:37.409134    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c889d49bb-fxp5r"]
Sep 17 19:13:38.452485 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:13:38.452436    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f08e542-7c5f-459d-970f-8cc3a34b3546" path="/var/lib/kubelet/pods/9f08e542-7c5f-459d-970f-8cc3a34b3546/volumes"
Sep 17 19:15:13.325724 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.325690    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"]
Sep 17 19:15:13.326225 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.326020    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f08e542-7c5f-459d-970f-8cc3a34b3546" containerName="console"
Sep 17 19:15:13.326225 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.326031    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f08e542-7c5f-459d-970f-8cc3a34b3546" containerName="console"
Sep 17 19:15:13.326225 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.326122    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f08e542-7c5f-459d-970f-8cc3a34b3546" containerName="console"
Sep 17 19:15:13.328215 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.328200    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:15:13.332715 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.332692    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-53780-predictor-serving-cert\""
Sep 17 19:15:13.333226 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.333206    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tmb8q\""
Sep 17 19:15:13.333328 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.333287    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:15:13.333328 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.333315    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:15:13.333435 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.333396    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-53780-kube-rbac-proxy-sar-config\""
Sep 17 19:15:13.356671 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.356646    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"]
Sep 17 19:15:13.433180 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.433149    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-53780-predictor-674f6fd797-jw9cv\" (UID: \"7c686f43-8f7a-4dae-b4d6-b74477b1fce2\") " pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:15:13.433346 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.433198    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-proxy-tls\") pod \"success-200-isvc-53780-predictor-674f6fd797-jw9cv\" (UID: \"7c686f43-8f7a-4dae-b4d6-b74477b1fce2\") " pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:15:13.433346 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.433278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvv8\" (UniqueName: \"kubernetes.io/projected/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-kube-api-access-5cvv8\") pod \"success-200-isvc-53780-predictor-674f6fd797-jw9cv\" (UID: \"7c686f43-8f7a-4dae-b4d6-b74477b1fce2\") " pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:15:13.511962 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.511930    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"]
Sep 17 19:15:13.514187 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.514172    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:15:13.516571 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.516550    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-53780-kube-rbac-proxy-sar-config\""
Sep 17 19:15:13.516704 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.516623    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-53780-predictor-serving-cert\""
Sep 17 19:15:13.523984 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.523963    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"]
Sep 17 19:15:13.533742 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.533721    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-proxy-tls\") pod \"success-200-isvc-53780-predictor-674f6fd797-jw9cv\" (UID: \"7c686f43-8f7a-4dae-b4d6-b74477b1fce2\") " pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:15:13.533845 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.533752    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/419f9c33-a1bc-4cec-9c3e-152642be1db8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-53780-predictor-c576bbf4d-7rn4m\" (UID: \"419f9c33-a1bc-4cec-9c3e-152642be1db8\") " pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:15:13.533845 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.533778    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419f9c33-a1bc-4cec-9c3e-152642be1db8-proxy-tls\") pod \"error-404-isvc-53780-predictor-c576bbf4d-7rn4m\" (UID: \"419f9c33-a1bc-4cec-9c3e-152642be1db8\") " pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:15:13.533845 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.533798    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzbwg\" (UniqueName: \"kubernetes.io/projected/419f9c33-a1bc-4cec-9c3e-152642be1db8-kube-api-access-lzbwg\") pod \"error-404-isvc-53780-predictor-c576bbf4d-7rn4m\" (UID: \"419f9c33-a1bc-4cec-9c3e-152642be1db8\") " pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:15:13.533971 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.533891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvv8\" (UniqueName: \"kubernetes.io/projected/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-kube-api-access-5cvv8\") pod \"success-200-isvc-53780-predictor-674f6fd797-jw9cv\" (UID: \"7c686f43-8f7a-4dae-b4d6-b74477b1fce2\") " pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:15:13.533971 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.533952    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-53780-predictor-674f6fd797-jw9cv\" (UID: \"7c686f43-8f7a-4dae-b4d6-b74477b1fce2\") " pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:15:13.534752 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.534732    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-53780-predictor-674f6fd797-jw9cv\" (UID: \"7c686f43-8f7a-4dae-b4d6-b74477b1fce2\") " pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:15:13.536116 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.536101    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-proxy-tls\") pod \"success-200-isvc-53780-predictor-674f6fd797-jw9cv\" (UID: \"7c686f43-8f7a-4dae-b4d6-b74477b1fce2\") " pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:15:13.544489 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.544435    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvv8\" (UniqueName: \"kubernetes.io/projected/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-kube-api-access-5cvv8\") pod \"success-200-isvc-53780-predictor-674f6fd797-jw9cv\" (UID: \"7c686f43-8f7a-4dae-b4d6-b74477b1fce2\") " pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:15:13.634734 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.634651    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/419f9c33-a1bc-4cec-9c3e-152642be1db8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-53780-predictor-c576bbf4d-7rn4m\" (UID: \"419f9c33-a1bc-4cec-9c3e-152642be1db8\") " pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:15:13.634734 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.634695    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419f9c33-a1bc-4cec-9c3e-152642be1db8-proxy-tls\") pod \"error-404-isvc-53780-predictor-c576bbf4d-7rn4m\" (UID: \"419f9c33-a1bc-4cec-9c3e-152642be1db8\") " pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:15:13.634918 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.634748    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzbwg\" (UniqueName: \"kubernetes.io/projected/419f9c33-a1bc-4cec-9c3e-152642be1db8-kube-api-access-lzbwg\") pod \"error-404-isvc-53780-predictor-c576bbf4d-7rn4m\" (UID: \"419f9c33-a1bc-4cec-9c3e-152642be1db8\") " pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:15:13.634966 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:15:13.634915    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-53780-predictor-serving-cert: secret "error-404-isvc-53780-predictor-serving-cert" not found
Sep 17 19:15:13.635007 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:15:13.634986    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419f9c33-a1bc-4cec-9c3e-152642be1db8-proxy-tls podName:419f9c33-a1bc-4cec-9c3e-152642be1db8 nodeName:}" failed. No retries permitted until 2026-09-17 19:15:14.134966337 +0000 UTC m=+517.962824566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/419f9c33-a1bc-4cec-9c3e-152642be1db8-proxy-tls") pod "error-404-isvc-53780-predictor-c576bbf4d-7rn4m" (UID: "419f9c33-a1bc-4cec-9c3e-152642be1db8") : secret "error-404-isvc-53780-predictor-serving-cert" not found
Sep 17 19:15:13.635420 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.635401    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/419f9c33-a1bc-4cec-9c3e-152642be1db8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-53780-predictor-c576bbf4d-7rn4m\" (UID: \"419f9c33-a1bc-4cec-9c3e-152642be1db8\") " pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:15:13.637437 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.637414    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:15:13.648918 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.648865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzbwg\" (UniqueName: \"kubernetes.io/projected/419f9c33-a1bc-4cec-9c3e-152642be1db8-kube-api-access-lzbwg\") pod \"error-404-isvc-53780-predictor-c576bbf4d-7rn4m\" (UID: \"419f9c33-a1bc-4cec-9c3e-152642be1db8\") " pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:15:13.913395 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:13.913369    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"]
Sep 17 19:15:13.915568 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:15:13.915539    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c686f43_8f7a_4dae_b4d6_b74477b1fce2.slice/crio-f5ab738807f91046cb0d76968d8096455a99ba282bf0a8a936f6db8975081790 WatchSource:0}: Error finding container f5ab738807f91046cb0d76968d8096455a99ba282bf0a8a936f6db8975081790: Status 404 returned error can't find the container with id f5ab738807f91046cb0d76968d8096455a99ba282bf0a8a936f6db8975081790
Sep 17 19:15:14.139052 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:14.139018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419f9c33-a1bc-4cec-9c3e-152642be1db8-proxy-tls\") pod \"error-404-isvc-53780-predictor-c576bbf4d-7rn4m\" (UID: \"419f9c33-a1bc-4cec-9c3e-152642be1db8\") " pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:15:14.141259 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:14.141239    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419f9c33-a1bc-4cec-9c3e-152642be1db8-proxy-tls\") pod \"error-404-isvc-53780-predictor-c576bbf4d-7rn4m\" (UID: \"419f9c33-a1bc-4cec-9c3e-152642be1db8\") " pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:15:14.423943 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:14.423910    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:15:14.758732 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:14.758556    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"]
Sep 17 19:15:14.765752 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:15:14.765714    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419f9c33_a1bc_4cec_9c3e_152642be1db8.slice/crio-621eb492f5270b4817f4f1d51518a58bad033f4c2687fe1c6aed221900713777 WatchSource:0}: Error finding container 621eb492f5270b4817f4f1d51518a58bad033f4c2687fe1c6aed221900713777: Status 404 returned error can't find the container with id 621eb492f5270b4817f4f1d51518a58bad033f4c2687fe1c6aed221900713777
Sep 17 19:15:14.820900 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:14.820860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv" event={"ID":"7c686f43-8f7a-4dae-b4d6-b74477b1fce2","Type":"ContainerStarted","Data":"f5ab738807f91046cb0d76968d8096455a99ba282bf0a8a936f6db8975081790"}
Sep 17 19:15:14.822776 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:14.822748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m" event={"ID":"419f9c33-a1bc-4cec-9c3e-152642be1db8","Type":"ContainerStarted","Data":"621eb492f5270b4817f4f1d51518a58bad033f4c2687fe1c6aed221900713777"}
Sep 17 19:15:29.935913 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:29.935806    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv" event={"ID":"7c686f43-8f7a-4dae-b4d6-b74477b1fce2","Type":"ContainerStarted","Data":"87e2eb27544c275df1a323391aca189f6dd6c07427a0491b1b7e01bf46c97e37"}
Sep 17 19:15:29.939266 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:29.939173    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m" event={"ID":"419f9c33-a1bc-4cec-9c3e-152642be1db8","Type":"ContainerStarted","Data":"ee82eb9e61de433ae37e441172badd5028091bf320fc287d8ab9f266d832fe65"}
Sep 17 19:15:32.958436 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:32.958393    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m" event={"ID":"419f9c33-a1bc-4cec-9c3e-152642be1db8","Type":"ContainerStarted","Data":"f0a31aa5424c98acfa96901265aa384adc2c62152eb9ca57cce39f8b32b3c039"}
Sep 17 19:15:32.958903 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:32.958495    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:15:32.958903 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:32.958526    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:15:32.960242 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:32.960212    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m" podUID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:15:32.960705 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:32.960687    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv" event={"ID":"7c686f43-8f7a-4dae-b4d6-b74477b1fce2","Type":"ContainerStarted","Data":"2b1026e4250f9eceec35d28d3d48575ba5d9195c19d4923dc0059861b0fe2a07"}
Sep 17 19:15:32.960803 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:32.960742    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:15:32.960803 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:32.960765    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:15:32.961735 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:32.961709    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv" podUID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 19:15:32.976870 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:32.976812    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m" podStartSLOduration=2.811489153 podStartE2EDuration="19.976796404s" totalImagesPullingTime="17.165307251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:15:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:15:32.975829248 +0000 UTC m=+536.803687473" watchObservedRunningTime="2026-09-17 19:15:32.976796404 +0000 UTC m=+536.804654637"
Sep 17 19:15:32.993704 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:32.993656    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv" podStartSLOduration=1.971930608 podStartE2EDuration="19.99364024s" totalImagesPullingTime="18.021709632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:15:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:15:32.992397366 +0000 UTC m=+536.820255600" watchObservedRunningTime="2026-09-17 19:15:32.99364024 +0000 UTC m=+536.821498473"
Sep 17 19:15:33.966867 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:33.966825    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv" podUID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 19:15:33.967251 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:33.966826    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m" podUID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:15:38.971863 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:38.971835    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:15:38.972475 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:38.972429    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m" podUID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:15:38.977725 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:38.977706    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:15:38.978078 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:38.978044    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv" podUID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 19:15:48.972503 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:48.972466    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m" podUID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:15:48.978689 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:48.978662    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv" podUID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 19:15:58.973252 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:58.973215    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m" podUID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:15:58.978522 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:15:58.978496    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv" podUID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 19:16:08.972972 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:08.972932    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m" podUID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 19:16:08.978297 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:08.978271    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv" podUID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 19:16:18.976619 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:18.976583    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:16:18.978578 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:18.978559    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:16:38.349956 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:38.349921    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:16:38.350355 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:38.349974    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:16:44.162717 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.162684    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"]
Sep 17 19:16:44.163113 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.162951    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv" podUID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerName="kserve-container" containerID="cri-o://87e2eb27544c275df1a323391aca189f6dd6c07427a0491b1b7e01bf46c97e37" gracePeriod=30
Sep 17 19:16:44.163113 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.162976    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv" podUID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerName="kube-rbac-proxy" containerID="cri-o://2b1026e4250f9eceec35d28d3d48575ba5d9195c19d4923dc0059861b0fe2a07" gracePeriod=30
Sep 17 19:16:44.231938 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.231906    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"]
Sep 17 19:16:44.234403 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.234387    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:16:44.237119 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.237088    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8c825-predictor-serving-cert\""
Sep 17 19:16:44.237221 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.237205    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8c825-kube-rbac-proxy-sar-config\""
Sep 17 19:16:44.245859 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.245835    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"]
Sep 17 19:16:44.281974 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.281941    2583 generic.go:356] "Generic (PLEG): container finished" podID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerID="2b1026e4250f9eceec35d28d3d48575ba5d9195c19d4923dc0059861b0fe2a07" exitCode=2
Sep 17 19:16:44.282130 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.281986    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv" event={"ID":"7c686f43-8f7a-4dae-b4d6-b74477b1fce2","Type":"ContainerDied","Data":"2b1026e4250f9eceec35d28d3d48575ba5d9195c19d4923dc0059861b0fe2a07"}
Sep 17 19:16:44.290221 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.290181    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"]
Sep 17 19:16:44.290407 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.290383    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m" podUID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerName="kserve-container" containerID="cri-o://ee82eb9e61de433ae37e441172badd5028091bf320fc287d8ab9f266d832fe65" gracePeriod=30
Sep 17 19:16:44.290502 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.290437    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m" podUID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerName="kube-rbac-proxy" containerID="cri-o://f0a31aa5424c98acfa96901265aa384adc2c62152eb9ca57cce39f8b32b3c039" gracePeriod=30
Sep 17 19:16:44.309751 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.309719    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3743c8e9-c568-4a36-9b7c-b4d1653ff448-proxy-tls\") pod \"success-200-isvc-8c825-predictor-67999bcd6c-r4ljh\" (UID: \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\") " pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:16:44.309895 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.309852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3743c8e9-c568-4a36-9b7c-b4d1653ff448-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8c825-predictor-67999bcd6c-r4ljh\" (UID: \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\") " pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:16:44.309895 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.309884    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5df7\" (UniqueName: \"kubernetes.io/projected/3743c8e9-c568-4a36-9b7c-b4d1653ff448-kube-api-access-s5df7\") pod \"success-200-isvc-8c825-predictor-67999bcd6c-r4ljh\" (UID: \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\") " pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:16:44.361941 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.361910    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"]
Sep 17 19:16:44.364482 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.364466    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:16:44.367169 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.367149    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8c825-predictor-serving-cert\""
Sep 17 19:16:44.367585 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.367565    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8c825-kube-rbac-proxy-sar-config\""
Sep 17 19:16:44.377491 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.377468    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"]
Sep 17 19:16:44.411110 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.411080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3743c8e9-c568-4a36-9b7c-b4d1653ff448-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8c825-predictor-67999bcd6c-r4ljh\" (UID: \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\") " pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:16:44.411247 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.411128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5df7\" (UniqueName: \"kubernetes.io/projected/3743c8e9-c568-4a36-9b7c-b4d1653ff448-kube-api-access-s5df7\") pod \"success-200-isvc-8c825-predictor-67999bcd6c-r4ljh\" (UID: \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\") " pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:16:44.411247 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.411196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3743c8e9-c568-4a36-9b7c-b4d1653ff448-proxy-tls\") pod \"success-200-isvc-8c825-predictor-67999bcd6c-r4ljh\" (UID: \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\") " pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:16:44.411363 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:16:44.411309    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-8c825-predictor-serving-cert: secret "success-200-isvc-8c825-predictor-serving-cert" not found
Sep 17 19:16:44.411416 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:16:44.411364    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3743c8e9-c568-4a36-9b7c-b4d1653ff448-proxy-tls podName:3743c8e9-c568-4a36-9b7c-b4d1653ff448 nodeName:}" failed. No retries permitted until 2026-09-17 19:16:44.911350405 +0000 UTC m=+608.739208617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3743c8e9-c568-4a36-9b7c-b4d1653ff448-proxy-tls") pod "success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" (UID: "3743c8e9-c568-4a36-9b7c-b4d1653ff448") : secret "success-200-isvc-8c825-predictor-serving-cert" not found
Sep 17 19:16:44.411765 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.411742    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3743c8e9-c568-4a36-9b7c-b4d1653ff448-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8c825-predictor-67999bcd6c-r4ljh\" (UID: \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\") " pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:16:44.423315 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.423249    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5df7\" (UniqueName: \"kubernetes.io/projected/3743c8e9-c568-4a36-9b7c-b4d1653ff448-kube-api-access-s5df7\") pod \"success-200-isvc-8c825-predictor-67999bcd6c-r4ljh\" (UID: \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\") " pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:16:44.512160 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.512121    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-proxy-tls\") pod \"error-404-isvc-8c825-predictor-7f9d986954-pxj22\" (UID: \"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8\") " pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:16:44.512340 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.512196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8c825-predictor-7f9d986954-pxj22\" (UID: \"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8\") " pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:16:44.512340 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.512260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksxmx\" (UniqueName: \"kubernetes.io/projected/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-kube-api-access-ksxmx\") pod \"error-404-isvc-8c825-predictor-7f9d986954-pxj22\" (UID: \"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8\") " pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:16:44.613249 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.613203    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-proxy-tls\") pod \"error-404-isvc-8c825-predictor-7f9d986954-pxj22\" (UID: \"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8\") " pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:16:44.613373 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.613287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8c825-predictor-7f9d986954-pxj22\" (UID: \"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8\") " pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:16:44.613373 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.613318    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksxmx\" (UniqueName: \"kubernetes.io/projected/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-kube-api-access-ksxmx\") pod \"error-404-isvc-8c825-predictor-7f9d986954-pxj22\" (UID: \"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8\") " pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:16:44.614112 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.614088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8c825-predictor-7f9d986954-pxj22\" (UID: \"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8\") " pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:16:44.615833 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.615803    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-proxy-tls\") pod \"error-404-isvc-8c825-predictor-7f9d986954-pxj22\" (UID: \"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8\") " pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:16:44.622671 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.622647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksxmx\" (UniqueName: \"kubernetes.io/projected/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-kube-api-access-ksxmx\") pod \"error-404-isvc-8c825-predictor-7f9d986954-pxj22\" (UID: \"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8\") " pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:16:44.673477 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.673369    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:16:44.916181 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.916124    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3743c8e9-c568-4a36-9b7c-b4d1653ff448-proxy-tls\") pod \"success-200-isvc-8c825-predictor-67999bcd6c-r4ljh\" (UID: \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\") " pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:16:44.918544 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.918520    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3743c8e9-c568-4a36-9b7c-b4d1653ff448-proxy-tls\") pod \"success-200-isvc-8c825-predictor-67999bcd6c-r4ljh\" (UID: \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\") " pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:16:44.943768 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:44.943744    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"]
Sep 17 19:16:44.945894 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:16:44.945860    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17cd3d4_707c_4fe7_8f28_7f8278f0abc8.slice/crio-d786a5f69abe1b126dea6da567acc7a280adeabf322dc86ce19689cc8f1ab05c WatchSource:0}: Error finding container d786a5f69abe1b126dea6da567acc7a280adeabf322dc86ce19689cc8f1ab05c: Status 404 returned error can't find the container with id d786a5f69abe1b126dea6da567acc7a280adeabf322dc86ce19689cc8f1ab05c
Sep 17 19:16:45.144795 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:45.144756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:16:45.288261 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:45.288228    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" event={"ID":"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8","Type":"ContainerStarted","Data":"2cd521e83fe30096ec72e4937b892152bf3c771e7771b72a57084bfd61501efe"}
Sep 17 19:16:45.288735 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:45.288295    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" event={"ID":"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8","Type":"ContainerStarted","Data":"6ade7e702b0f33ade11d645bc7b5c4cc46024da3cc68fdf0d33246d20884eb73"}
Sep 17 19:16:45.288735 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:45.288329    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" event={"ID":"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8","Type":"ContainerStarted","Data":"d786a5f69abe1b126dea6da567acc7a280adeabf322dc86ce19689cc8f1ab05c"}
Sep 17 19:16:45.288735 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:45.288358    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:16:45.288735 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:45.288380    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:16:45.290125 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:45.290094    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:16:45.290239 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:45.290219    2583 generic.go:356] "Generic (PLEG): container finished" podID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerID="f0a31aa5424c98acfa96901265aa384adc2c62152eb9ca57cce39f8b32b3c039" exitCode=2
Sep 17 19:16:45.290296 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:45.290271    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m" event={"ID":"419f9c33-a1bc-4cec-9c3e-152642be1db8","Type":"ContainerDied","Data":"f0a31aa5424c98acfa96901265aa384adc2c62152eb9ca57cce39f8b32b3c039"}
Sep 17 19:16:45.308239 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:45.308183    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" podStartSLOduration=1.308165536 podStartE2EDuration="1.308165536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:16:45.307095447 +0000 UTC m=+609.134953680" watchObservedRunningTime="2026-09-17 19:16:45.308165536 +0000 UTC m=+609.136023769"
Sep 17 19:16:45.412872 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:45.412848    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"]
Sep 17 19:16:45.415453 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:16:45.415409    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3743c8e9_c568_4a36_9b7c_b4d1653ff448.slice/crio-aa8d8da0beacf6fc846cd8aed872a53dd92fa9ee51e2478df1bc99e068382eeb WatchSource:0}: Error finding container aa8d8da0beacf6fc846cd8aed872a53dd92fa9ee51e2478df1bc99e068382eeb: Status 404 returned error can't find the container with id aa8d8da0beacf6fc846cd8aed872a53dd92fa9ee51e2478df1bc99e068382eeb
Sep 17 19:16:46.296599 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:46.296566    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" event={"ID":"3743c8e9-c568-4a36-9b7c-b4d1653ff448","Type":"ContainerStarted","Data":"3a67000eedf150ac33674843d87dce5fef14cf4c84fc15483046c4c0561d84c5"}
Sep 17 19:16:46.296985 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:46.296623    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" event={"ID":"3743c8e9-c568-4a36-9b7c-b4d1653ff448","Type":"ContainerStarted","Data":"ba8a7e1f75a241e9790a69e270ec0e24a9f3f410dc47f81ef01c7d2328f42571"}
Sep 17 19:16:46.296985 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:46.296645    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" event={"ID":"3743c8e9-c568-4a36-9b7c-b4d1653ff448","Type":"ContainerStarted","Data":"aa8d8da0beacf6fc846cd8aed872a53dd92fa9ee51e2478df1bc99e068382eeb"}
Sep 17 19:16:46.297060 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:46.296981    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:16:46.317509 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:46.317429    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" podStartSLOduration=2.317406593 podStartE2EDuration="2.317406593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:16:46.31402078 +0000 UTC m=+610.141879013" watchObservedRunningTime="2026-09-17 19:16:46.317406593 +0000 UTC m=+610.145264828"
Sep 17 19:16:47.302031 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:47.302000    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:16:47.302413 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:47.302085    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:16:47.303492 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:47.303465    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:16:48.017317 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.017283    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:16:48.146170 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.146147    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:16:48.148159 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.148139    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-proxy-tls\") pod \"7c686f43-8f7a-4dae-b4d6-b74477b1fce2\" (UID: \"7c686f43-8f7a-4dae-b4d6-b74477b1fce2\") "
Sep 17 19:16:48.148294 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.148224    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cvv8\" (UniqueName: \"kubernetes.io/projected/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-kube-api-access-5cvv8\") pod \"7c686f43-8f7a-4dae-b4d6-b74477b1fce2\" (UID: \"7c686f43-8f7a-4dae-b4d6-b74477b1fce2\") "
Sep 17 19:16:48.148294 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.148253    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-kube-rbac-proxy-sar-config\") pod \"7c686f43-8f7a-4dae-b4d6-b74477b1fce2\" (UID: \"7c686f43-8f7a-4dae-b4d6-b74477b1fce2\") "
Sep 17 19:16:48.148639 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.148617    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c686f43-8f7a-4dae-b4d6-b74477b1fce2" (UID: "7c686f43-8f7a-4dae-b4d6-b74477b1fce2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:16:48.150253 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.150231    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c686f43-8f7a-4dae-b4d6-b74477b1fce2" (UID: "7c686f43-8f7a-4dae-b4d6-b74477b1fce2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:16:48.150350 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.150337    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-kube-api-access-5cvv8" (OuterVolumeSpecName: "kube-api-access-5cvv8") pod "7c686f43-8f7a-4dae-b4d6-b74477b1fce2" (UID: "7c686f43-8f7a-4dae-b4d6-b74477b1fce2"). InnerVolumeSpecName "kube-api-access-5cvv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:16:48.248862 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.248822    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/419f9c33-a1bc-4cec-9c3e-152642be1db8-kube-rbac-proxy-sar-config\") pod \"419f9c33-a1bc-4cec-9c3e-152642be1db8\" (UID: \"419f9c33-a1bc-4cec-9c3e-152642be1db8\") "
Sep 17 19:16:48.248862 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.248873    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419f9c33-a1bc-4cec-9c3e-152642be1db8-proxy-tls\") pod \"419f9c33-a1bc-4cec-9c3e-152642be1db8\" (UID: \"419f9c33-a1bc-4cec-9c3e-152642be1db8\") "
Sep 17 19:16:48.249068 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.248927    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzbwg\" (UniqueName: \"kubernetes.io/projected/419f9c33-a1bc-4cec-9c3e-152642be1db8-kube-api-access-lzbwg\") pod \"419f9c33-a1bc-4cec-9c3e-152642be1db8\" (UID: \"419f9c33-a1bc-4cec-9c3e-152642be1db8\") "
Sep 17 19:16:48.249138 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.249121    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cvv8\" (UniqueName: \"kubernetes.io/projected/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-kube-api-access-5cvv8\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:16:48.249183 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.249144    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:16:48.249183 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.249158    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c686f43-8f7a-4dae-b4d6-b74477b1fce2-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:16:48.249261 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.249182    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/419f9c33-a1bc-4cec-9c3e-152642be1db8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "419f9c33-a1bc-4cec-9c3e-152642be1db8" (UID: "419f9c33-a1bc-4cec-9c3e-152642be1db8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:16:48.250969 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.250944    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419f9c33-a1bc-4cec-9c3e-152642be1db8-kube-api-access-lzbwg" (OuterVolumeSpecName: "kube-api-access-lzbwg") pod "419f9c33-a1bc-4cec-9c3e-152642be1db8" (UID: "419f9c33-a1bc-4cec-9c3e-152642be1db8"). InnerVolumeSpecName "kube-api-access-lzbwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:16:48.251027 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.251006    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/419f9c33-a1bc-4cec-9c3e-152642be1db8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "419f9c33-a1bc-4cec-9c3e-152642be1db8" (UID: "419f9c33-a1bc-4cec-9c3e-152642be1db8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:16:48.308079 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.308047    2583 generic.go:356] "Generic (PLEG): container finished" podID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerID="ee82eb9e61de433ae37e441172badd5028091bf320fc287d8ab9f266d832fe65" exitCode=0
Sep 17 19:16:48.308546 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.308096    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"
Sep 17 19:16:48.308546 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.308130    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m" event={"ID":"419f9c33-a1bc-4cec-9c3e-152642be1db8","Type":"ContainerDied","Data":"ee82eb9e61de433ae37e441172badd5028091bf320fc287d8ab9f266d832fe65"}
Sep 17 19:16:48.308546 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.308199    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m" event={"ID":"419f9c33-a1bc-4cec-9c3e-152642be1db8","Type":"ContainerDied","Data":"621eb492f5270b4817f4f1d51518a58bad033f4c2687fe1c6aed221900713777"}
Sep 17 19:16:48.308546 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.308239    2583 scope.go:117] "RemoveContainer" containerID="f0a31aa5424c98acfa96901265aa384adc2c62152eb9ca57cce39f8b32b3c039"
Sep 17 19:16:48.309800 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.309777    2583 generic.go:356] "Generic (PLEG): container finished" podID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerID="87e2eb27544c275df1a323391aca189f6dd6c07427a0491b1b7e01bf46c97e37" exitCode=0
Sep 17 19:16:48.309918 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.309832    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"
Sep 17 19:16:48.309918 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.309884    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv" event={"ID":"7c686f43-8f7a-4dae-b4d6-b74477b1fce2","Type":"ContainerDied","Data":"87e2eb27544c275df1a323391aca189f6dd6c07427a0491b1b7e01bf46c97e37"}
Sep 17 19:16:48.310020 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.309947    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv" event={"ID":"7c686f43-8f7a-4dae-b4d6-b74477b1fce2","Type":"ContainerDied","Data":"f5ab738807f91046cb0d76968d8096455a99ba282bf0a8a936f6db8975081790"}
Sep 17 19:16:48.310360 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.310318    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:16:48.318014 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.317986    2583 scope.go:117] "RemoveContainer" containerID="ee82eb9e61de433ae37e441172badd5028091bf320fc287d8ab9f266d832fe65"
Sep 17 19:16:48.326664 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.326644    2583 scope.go:117] "RemoveContainer" containerID="f0a31aa5424c98acfa96901265aa384adc2c62152eb9ca57cce39f8b32b3c039"
Sep 17 19:16:48.326979 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:16:48.326958    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0a31aa5424c98acfa96901265aa384adc2c62152eb9ca57cce39f8b32b3c039\": container with ID starting with f0a31aa5424c98acfa96901265aa384adc2c62152eb9ca57cce39f8b32b3c039 not found: ID does not exist" containerID="f0a31aa5424c98acfa96901265aa384adc2c62152eb9ca57cce39f8b32b3c039"
Sep 17 19:16:48.327023 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.326989    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a31aa5424c98acfa96901265aa384adc2c62152eb9ca57cce39f8b32b3c039"} err="failed to get container status \"f0a31aa5424c98acfa96901265aa384adc2c62152eb9ca57cce39f8b32b3c039\": rpc error: code = NotFound desc = could not find container \"f0a31aa5424c98acfa96901265aa384adc2c62152eb9ca57cce39f8b32b3c039\": container with ID starting with f0a31aa5424c98acfa96901265aa384adc2c62152eb9ca57cce39f8b32b3c039 not found: ID does not exist"
Sep 17 19:16:48.327023 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.327006    2583 scope.go:117] "RemoveContainer" containerID="ee82eb9e61de433ae37e441172badd5028091bf320fc287d8ab9f266d832fe65"
Sep 17 19:16:48.327249 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:16:48.327234    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee82eb9e61de433ae37e441172badd5028091bf320fc287d8ab9f266d832fe65\": container with ID starting with ee82eb9e61de433ae37e441172badd5028091bf320fc287d8ab9f266d832fe65 not found: ID does not exist" containerID="ee82eb9e61de433ae37e441172badd5028091bf320fc287d8ab9f266d832fe65"
Sep 17 19:16:48.327286 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.327254    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee82eb9e61de433ae37e441172badd5028091bf320fc287d8ab9f266d832fe65"} err="failed to get container status \"ee82eb9e61de433ae37e441172badd5028091bf320fc287d8ab9f266d832fe65\": rpc error: code = NotFound desc = could not find container \"ee82eb9e61de433ae37e441172badd5028091bf320fc287d8ab9f266d832fe65\": container with ID starting with ee82eb9e61de433ae37e441172badd5028091bf320fc287d8ab9f266d832fe65 not found: ID does not exist"
Sep 17 19:16:48.327286 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.327268    2583 scope.go:117] "RemoveContainer" containerID="2b1026e4250f9eceec35d28d3d48575ba5d9195c19d4923dc0059861b0fe2a07"
Sep 17 19:16:48.334751 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.334723    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"]
Sep 17 19:16:48.335782 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.335765    2583 scope.go:117] "RemoveContainer" containerID="87e2eb27544c275df1a323391aca189f6dd6c07427a0491b1b7e01bf46c97e37"
Sep 17 19:16:48.337328 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.337309    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53780-predictor-c576bbf4d-7rn4m"]
Sep 17 19:16:48.343347 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.343325    2583 scope.go:117] "RemoveContainer" containerID="2b1026e4250f9eceec35d28d3d48575ba5d9195c19d4923dc0059861b0fe2a07"
Sep 17 19:16:48.343658 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:16:48.343638    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1026e4250f9eceec35d28d3d48575ba5d9195c19d4923dc0059861b0fe2a07\": container with ID starting with 2b1026e4250f9eceec35d28d3d48575ba5d9195c19d4923dc0059861b0fe2a07 not found: ID does not exist" containerID="2b1026e4250f9eceec35d28d3d48575ba5d9195c19d4923dc0059861b0fe2a07"
Sep 17 19:16:48.343720 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.343666    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1026e4250f9eceec35d28d3d48575ba5d9195c19d4923dc0059861b0fe2a07"} err="failed to get container status \"2b1026e4250f9eceec35d28d3d48575ba5d9195c19d4923dc0059861b0fe2a07\": rpc error: code = NotFound desc = could not find container \"2b1026e4250f9eceec35d28d3d48575ba5d9195c19d4923dc0059861b0fe2a07\": container with ID starting with 2b1026e4250f9eceec35d28d3d48575ba5d9195c19d4923dc0059861b0fe2a07 not found: ID does not exist"
Sep 17 19:16:48.343720 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.343684    2583 scope.go:117] "RemoveContainer" containerID="87e2eb27544c275df1a323391aca189f6dd6c07427a0491b1b7e01bf46c97e37"
Sep 17 19:16:48.343927 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:16:48.343914    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e2eb27544c275df1a323391aca189f6dd6c07427a0491b1b7e01bf46c97e37\": container with ID starting with 87e2eb27544c275df1a323391aca189f6dd6c07427a0491b1b7e01bf46c97e37 not found: ID does not exist" containerID="87e2eb27544c275df1a323391aca189f6dd6c07427a0491b1b7e01bf46c97e37"
Sep 17 19:16:48.343965 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.343928    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e2eb27544c275df1a323391aca189f6dd6c07427a0491b1b7e01bf46c97e37"} err="failed to get container status \"87e2eb27544c275df1a323391aca189f6dd6c07427a0491b1b7e01bf46c97e37\": rpc error: code = NotFound desc = could not find container \"87e2eb27544c275df1a323391aca189f6dd6c07427a0491b1b7e01bf46c97e37\": container with ID starting with 87e2eb27544c275df1a323391aca189f6dd6c07427a0491b1b7e01bf46c97e37 not found: ID does not exist"
Sep 17 19:16:48.347492 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.347463    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"]
Sep 17 19:16:48.349623 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.349603    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzbwg\" (UniqueName: \"kubernetes.io/projected/419f9c33-a1bc-4cec-9c3e-152642be1db8-kube-api-access-lzbwg\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:16:48.349750 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.349730    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/419f9c33-a1bc-4cec-9c3e-152642be1db8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:16:48.349750 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.349752    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/419f9c33-a1bc-4cec-9c3e-152642be1db8-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:16:48.351195 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.351177    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53780-predictor-674f6fd797-jw9cv"]
Sep 17 19:16:48.452010 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.451937    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="419f9c33-a1bc-4cec-9c3e-152642be1db8" path="/var/lib/kubelet/pods/419f9c33-a1bc-4cec-9c3e-152642be1db8/volumes"
Sep 17 19:16:48.452328 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:48.452317    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" path="/var/lib/kubelet/pods/7c686f43-8f7a-4dae-b4d6-b74477b1fce2/volumes"
Sep 17 19:16:51.301531 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:51.301493    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:16:51.302057 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:51.301903    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:16:53.314670 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:53.314639    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:16:53.315351 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:16:53.315322    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:17:01.302512 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:01.302474    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:17:03.315664 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:03.315616    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:17:11.302678 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:11.302639    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:17:13.315697 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:13.315656    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:17:21.302260 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:21.302219    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:17:23.316154 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:23.316097    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:17:31.302676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:31.302630    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:17:33.315702 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:33.315661    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 19:17:41.303523 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:41.303491    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:17:43.316622 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:43.316583    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:17:44.873005 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.872966    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"]
Sep 17 19:17:44.873381 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.873331    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerName="kserve-container"
Sep 17 19:17:44.873381 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.873342    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerName="kserve-container"
Sep 17 19:17:44.873381 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.873354    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerName="kube-rbac-proxy"
Sep 17 19:17:44.873381 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.873361    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerName="kube-rbac-proxy"
Sep 17 19:17:44.873381 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.873372    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerName="kserve-container"
Sep 17 19:17:44.873381 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.873378    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerName="kserve-container"
Sep 17 19:17:44.873593 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.873387    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerName="kube-rbac-proxy"
Sep 17 19:17:44.873593 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.873393    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerName="kube-rbac-proxy"
Sep 17 19:17:44.873593 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.873462    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerName="kserve-container"
Sep 17 19:17:44.873593 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.873471    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c686f43-8f7a-4dae-b4d6-b74477b1fce2" containerName="kube-rbac-proxy"
Sep 17 19:17:44.873593 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.873479    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerName="kube-rbac-proxy"
Sep 17 19:17:44.873593 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.873484    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="419f9c33-a1bc-4cec-9c3e-152642be1db8" containerName="kserve-container"
Sep 17 19:17:44.877620 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.877603    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:17:44.880150 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.880121    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fcbf3-predictor-serving-cert\""
Sep 17 19:17:44.880293 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.880206    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fcbf3-kube-rbac-proxy-sar-config\""
Sep 17 19:17:44.888818 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.888795    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"]
Sep 17 19:17:44.991059 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.991028    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"]
Sep 17 19:17:44.994376 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.994360    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:17:44.997143 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.997125    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fcbf3-kube-rbac-proxy-sar-config\""
Sep 17 19:17:44.997197 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:44.997143    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fcbf3-predictor-serving-cert\""
Sep 17 19:17:45.008058 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.008032    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"]
Sep 17 19:17:45.041235 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.041205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-proxy-tls\") pod \"success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh\" (UID: \"600dbcf2-78cf-4eb9-b8b5-95ff97b91539\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:17:45.041403 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.041258    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm79t\" (UniqueName: \"kubernetes.io/projected/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-kube-api-access-qm79t\") pod \"success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh\" (UID: \"600dbcf2-78cf-4eb9-b8b5-95ff97b91539\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:17:45.041403 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.041320    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh\" (UID: \"600dbcf2-78cf-4eb9-b8b5-95ff97b91539\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:17:45.142632 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.142546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh\" (UID: \"600dbcf2-78cf-4eb9-b8b5-95ff97b91539\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:17:45.142632 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.142618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctk6r\" (UniqueName: \"kubernetes.io/projected/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-kube-api-access-ctk6r\") pod \"error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs\" (UID: \"3bc44fcf-6903-4ce1-988d-2d66fe1da04d\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:17:45.142800 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.142659    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-proxy-tls\") pod \"error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs\" (UID: \"3bc44fcf-6903-4ce1-988d-2d66fe1da04d\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:17:45.142800 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.142701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-proxy-tls\") pod \"success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh\" (UID: \"600dbcf2-78cf-4eb9-b8b5-95ff97b91539\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:17:45.142800 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.142729    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs\" (UID: \"3bc44fcf-6903-4ce1-988d-2d66fe1da04d\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:17:45.142800 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.142768    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm79t\" (UniqueName: \"kubernetes.io/projected/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-kube-api-access-qm79t\") pod \"success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh\" (UID: \"600dbcf2-78cf-4eb9-b8b5-95ff97b91539\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:17:45.143240 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.143220    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh\" (UID: \"600dbcf2-78cf-4eb9-b8b5-95ff97b91539\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:17:45.145119 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.145098    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-proxy-tls\") pod \"success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh\" (UID: \"600dbcf2-78cf-4eb9-b8b5-95ff97b91539\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:17:45.150782 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.150764    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm79t\" (UniqueName: \"kubernetes.io/projected/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-kube-api-access-qm79t\") pod \"success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh\" (UID: \"600dbcf2-78cf-4eb9-b8b5-95ff97b91539\") " pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:17:45.187785 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.187749    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:17:45.244085 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.244057    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctk6r\" (UniqueName: \"kubernetes.io/projected/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-kube-api-access-ctk6r\") pod \"error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs\" (UID: \"3bc44fcf-6903-4ce1-988d-2d66fe1da04d\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:17:45.244219 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.244105    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-proxy-tls\") pod \"error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs\" (UID: \"3bc44fcf-6903-4ce1-988d-2d66fe1da04d\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:17:45.244219 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.244151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs\" (UID: \"3bc44fcf-6903-4ce1-988d-2d66fe1da04d\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:17:45.245022 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.244999    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs\" (UID: \"3bc44fcf-6903-4ce1-988d-2d66fe1da04d\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:17:45.249919 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.249855    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-proxy-tls\") pod \"error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs\" (UID: \"3bc44fcf-6903-4ce1-988d-2d66fe1da04d\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:17:45.256971 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.256944    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctk6r\" (UniqueName: \"kubernetes.io/projected/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-kube-api-access-ctk6r\") pod \"error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs\" (UID: \"3bc44fcf-6903-4ce1-988d-2d66fe1da04d\") " pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:17:45.304853 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.304812    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:17:45.458053 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.458031    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"]
Sep 17 19:17:45.459991 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:17:45.459970    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600dbcf2_78cf_4eb9_b8b5_95ff97b91539.slice/crio-66eb5a3ee8f90a595d25ae3c7f6ef854fab727482a22baa6e9c284647ad2047a WatchSource:0}: Error finding container 66eb5a3ee8f90a595d25ae3c7f6ef854fab727482a22baa6e9c284647ad2047a: Status 404 returned error can't find the container with id 66eb5a3ee8f90a595d25ae3c7f6ef854fab727482a22baa6e9c284647ad2047a
Sep 17 19:17:45.578552 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.578523    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"]
Sep 17 19:17:45.589474 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.589426    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" event={"ID":"600dbcf2-78cf-4eb9-b8b5-95ff97b91539","Type":"ContainerStarted","Data":"ec4e52e57f1f9249672f54d5e4a3c1415d566065fb01db39ebbc5440fa356d10"}
Sep 17 19:17:45.589577 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.589539    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" event={"ID":"600dbcf2-78cf-4eb9-b8b5-95ff97b91539","Type":"ContainerStarted","Data":"963cf20f90f326a14b23d12e3fdab58c989e20927a891939b0237cda59782937"}
Sep 17 19:17:45.589577 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.589572    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" event={"ID":"600dbcf2-78cf-4eb9-b8b5-95ff97b91539","Type":"ContainerStarted","Data":"66eb5a3ee8f90a595d25ae3c7f6ef854fab727482a22baa6e9c284647ad2047a"}
Sep 17 19:17:45.589674 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.589607    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:17:45.610757 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:45.610708    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" podStartSLOduration=1.610692738 podStartE2EDuration="1.610692738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:17:45.609271721 +0000 UTC m=+669.437129955" watchObservedRunningTime="2026-09-17 19:17:45.610692738 +0000 UTC m=+669.438550972"
Sep 17 19:17:46.596330 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:46.596292    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" event={"ID":"3bc44fcf-6903-4ce1-988d-2d66fe1da04d","Type":"ContainerStarted","Data":"62a28ead310dc6418def74e41a178aa22b836e81dfd6f8db51a3461227cf9764"}
Sep 17 19:17:46.596754 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:46.596361    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" event={"ID":"3bc44fcf-6903-4ce1-988d-2d66fe1da04d","Type":"ContainerStarted","Data":"0a210b7928ebb2b38958150f89480f242be94ee0d75b8a7a2458befd567bde7e"}
Sep 17 19:17:46.596754 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:46.596392    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" event={"ID":"3bc44fcf-6903-4ce1-988d-2d66fe1da04d","Type":"ContainerStarted","Data":"3cbfe3e9a4407f82db8d59e18247034d0f9fea6192dfec7ec0df7139a25ed055"}
Sep 17 19:17:46.596754 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:46.596655    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:17:46.598188 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:46.598163    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:17:46.618130 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:46.618078    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" podStartSLOduration=2.618062308 podStartE2EDuration="2.618062308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:17:46.617115821 +0000 UTC m=+670.444974070" watchObservedRunningTime="2026-09-17 19:17:46.618062308 +0000 UTC m=+670.445920521"
Sep 17 19:17:47.601404 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:47.601363    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:17:47.601875 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:47.601406    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:17:47.601875 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:47.601491    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:17:47.602945 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:47.602920    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:17:48.606261 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:48.606218    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:17:52.606691 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:52.606660    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:17:52.607305 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:52.607276    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:17:53.611480 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:53.611427    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:17:53.612010 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:17:53.611981    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:18:02.607687 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:18:02.607649    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:18:03.611897 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:18:03.611860    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:18:12.607827 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:18:12.607786    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:18:13.612868 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:18:13.612830    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:18:22.607751 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:18:22.607667    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:18:23.612859 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:18:23.612819    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:18:32.607870 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:18:32.607825    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:18:33.612855 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:18:33.612814    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:18:42.608731 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:18:42.608701    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:18:43.613643 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:18:43.613614    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:21:38.382892 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:21:38.382867    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:21:38.383781 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:21:38.383761    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:26:09.358819 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.358782    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"]
Sep 17 19:26:09.359314 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.359127    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerName="kserve-container" containerID="cri-o://ba8a7e1f75a241e9790a69e270ec0e24a9f3f410dc47f81ef01c7d2328f42571" gracePeriod=30
Sep 17 19:26:09.359314 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.359206    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerName="kube-rbac-proxy" containerID="cri-o://3a67000eedf150ac33674843d87dce5fef14cf4c84fc15483046c4c0561d84c5" gracePeriod=30
Sep 17 19:26:09.396866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.396832    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"]
Sep 17 19:26:09.397074 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.397054    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kserve-container" containerID="cri-o://6ade7e702b0f33ade11d645bc7b5c4cc46024da3cc68fdf0d33246d20884eb73" gracePeriod=30
Sep 17 19:26:09.397147 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.397121    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kube-rbac-proxy" containerID="cri-o://2cd521e83fe30096ec72e4937b892152bf3c771e7771b72a57084bfd61501efe" gracePeriod=30
Sep 17 19:26:09.521027 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.520988    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"]
Sep 17 19:26:09.524422 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.524398    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:26:09.526780 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.526756    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-87c35-predictor-serving-cert\""
Sep 17 19:26:09.527143 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.527120    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-87c35-kube-rbac-proxy-sar-config\""
Sep 17 19:26:09.538390 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.538365    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"]
Sep 17 19:26:09.582032 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.582000    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"]
Sep 17 19:26:09.585528 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.585508    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:26:09.588085 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.588064    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-87c35-predictor-serving-cert\""
Sep 17 19:26:09.588175 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.588068    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-87c35-kube-rbac-proxy-sar-config\""
Sep 17 19:26:09.595199 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.595174    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"]
Sep 17 19:26:09.663143 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.663051    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/590eb23d-6765-47e8-8bcd-9e96e06e3e55-proxy-tls\") pod \"success-200-isvc-87c35-predictor-77c4c6b6d7-pph49\" (UID: \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\") " pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:26:09.663143 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.663127    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrr5g\" (UniqueName: \"kubernetes.io/projected/590eb23d-6765-47e8-8bcd-9e96e06e3e55-kube-api-access-rrr5g\") pod \"success-200-isvc-87c35-predictor-77c4c6b6d7-pph49\" (UID: \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\") " pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:26:09.663358 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.663177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/590eb23d-6765-47e8-8bcd-9e96e06e3e55-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-87c35-predictor-77c4c6b6d7-pph49\" (UID: \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\") " pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:26:09.764032 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.763997    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf549\" (UniqueName: \"kubernetes.io/projected/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-kube-api-access-pf549\") pod \"error-404-isvc-87c35-predictor-586c54bbcd-cdtvn\" (UID: \"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3\") " pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:26:09.764231 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.764062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrr5g\" (UniqueName: \"kubernetes.io/projected/590eb23d-6765-47e8-8bcd-9e96e06e3e55-kube-api-access-rrr5g\") pod \"success-200-isvc-87c35-predictor-77c4c6b6d7-pph49\" (UID: \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\") " pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:26:09.764231 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.764086    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-proxy-tls\") pod \"error-404-isvc-87c35-predictor-586c54bbcd-cdtvn\" (UID: \"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3\") " pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:26:09.764231 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.764110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/590eb23d-6765-47e8-8bcd-9e96e06e3e55-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-87c35-predictor-77c4c6b6d7-pph49\" (UID: \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\") " pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:26:09.764231 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.764136    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-87c35-predictor-586c54bbcd-cdtvn\" (UID: \"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3\") " pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:26:09.764231 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.764209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/590eb23d-6765-47e8-8bcd-9e96e06e3e55-proxy-tls\") pod \"success-200-isvc-87c35-predictor-77c4c6b6d7-pph49\" (UID: \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\") " pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:26:09.764480 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:26:09.764299    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-87c35-predictor-serving-cert: secret "success-200-isvc-87c35-predictor-serving-cert" not found
Sep 17 19:26:09.764480 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:26:09.764367    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/590eb23d-6765-47e8-8bcd-9e96e06e3e55-proxy-tls podName:590eb23d-6765-47e8-8bcd-9e96e06e3e55 nodeName:}" failed. No retries permitted until 2026-09-17 19:26:10.264352402 +0000 UTC m=+1174.092210627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/590eb23d-6765-47e8-8bcd-9e96e06e3e55-proxy-tls") pod "success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" (UID: "590eb23d-6765-47e8-8bcd-9e96e06e3e55") : secret "success-200-isvc-87c35-predictor-serving-cert" not found
Sep 17 19:26:09.764754 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.764735    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/590eb23d-6765-47e8-8bcd-9e96e06e3e55-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-87c35-predictor-77c4c6b6d7-pph49\" (UID: \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\") " pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:26:09.775272 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.775243    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrr5g\" (UniqueName: \"kubernetes.io/projected/590eb23d-6765-47e8-8bcd-9e96e06e3e55-kube-api-access-rrr5g\") pod \"success-200-isvc-87c35-predictor-77c4c6b6d7-pph49\" (UID: \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\") " pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:26:09.865214 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.865178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-proxy-tls\") pod \"error-404-isvc-87c35-predictor-586c54bbcd-cdtvn\" (UID: \"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3\") " pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:26:09.865415 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.865225    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-87c35-predictor-586c54bbcd-cdtvn\" (UID: \"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3\") " pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:26:09.865415 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.865275    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf549\" (UniqueName: \"kubernetes.io/projected/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-kube-api-access-pf549\") pod \"error-404-isvc-87c35-predictor-586c54bbcd-cdtvn\" (UID: \"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3\") " pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:26:09.866013 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.865989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-87c35-predictor-586c54bbcd-cdtvn\" (UID: \"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3\") " pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:26:09.867632 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.867614    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-proxy-tls\") pod \"error-404-isvc-87c35-predictor-586c54bbcd-cdtvn\" (UID: \"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3\") " pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:26:09.873727 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.873702    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf549\" (UniqueName: \"kubernetes.io/projected/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-kube-api-access-pf549\") pod \"error-404-isvc-87c35-predictor-586c54bbcd-cdtvn\" (UID: \"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3\") " pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:26:09.895575 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.895539    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:26:09.914903 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.914825    2583 generic.go:356] "Generic (PLEG): container finished" podID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerID="2cd521e83fe30096ec72e4937b892152bf3c771e7771b72a57084bfd61501efe" exitCode=2
Sep 17 19:26:09.915032 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.914902    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" event={"ID":"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8","Type":"ContainerDied","Data":"2cd521e83fe30096ec72e4937b892152bf3c771e7771b72a57084bfd61501efe"}
Sep 17 19:26:09.917178 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.917151    2583 generic.go:356] "Generic (PLEG): container finished" podID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerID="3a67000eedf150ac33674843d87dce5fef14cf4c84fc15483046c4c0561d84c5" exitCode=2
Sep 17 19:26:09.917293 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:09.917232    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" event={"ID":"3743c8e9-c568-4a36-9b7c-b4d1653ff448","Type":"ContainerDied","Data":"3a67000eedf150ac33674843d87dce5fef14cf4c84fc15483046c4c0561d84c5"}
Sep 17 19:26:10.168797 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:10.168773    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"]
Sep 17 19:26:10.170605 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:26:10.170581    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc801e156_bc4a_4c8f_8a91_bce79f1ca6b3.slice/crio-fc9a1af7ac3068626b50b2af6bca212e94a559877c3ee8eafafd8d8e7cdbfa4c WatchSource:0}: Error finding container fc9a1af7ac3068626b50b2af6bca212e94a559877c3ee8eafafd8d8e7cdbfa4c: Status 404 returned error can't find the container with id fc9a1af7ac3068626b50b2af6bca212e94a559877c3ee8eafafd8d8e7cdbfa4c
Sep 17 19:26:10.268572 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:10.268542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/590eb23d-6765-47e8-8bcd-9e96e06e3e55-proxy-tls\") pod \"success-200-isvc-87c35-predictor-77c4c6b6d7-pph49\" (UID: \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\") " pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:26:10.270997 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:10.270956    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/590eb23d-6765-47e8-8bcd-9e96e06e3e55-proxy-tls\") pod \"success-200-isvc-87c35-predictor-77c4c6b6d7-pph49\" (UID: \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\") " pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:26:10.435894 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:10.435797    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:26:10.708944 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:10.708908    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"]
Sep 17 19:26:10.711055 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:26:10.711029    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590eb23d_6765_47e8_8bcd_9e96e06e3e55.slice/crio-c48c8bbd35c45dd384b04ff63ddbb7608c185850a7899efa4592865f131e2c79 WatchSource:0}: Error finding container c48c8bbd35c45dd384b04ff63ddbb7608c185850a7899efa4592865f131e2c79: Status 404 returned error can't find the container with id c48c8bbd35c45dd384b04ff63ddbb7608c185850a7899efa4592865f131e2c79
Sep 17 19:26:10.926519 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:10.926483    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" event={"ID":"590eb23d-6765-47e8-8bcd-9e96e06e3e55","Type":"ContainerStarted","Data":"e274e022aac0f2820ce63b94c8d2ae0fa69764a097d48c8b710386f2df9c53a6"}
Sep 17 19:26:10.926692 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:10.926550    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" event={"ID":"590eb23d-6765-47e8-8bcd-9e96e06e3e55","Type":"ContainerStarted","Data":"650c6fee146cb65eca0c412e83ced4ae82a2f1fa669824fd465da879017be185"}
Sep 17 19:26:10.926692 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:10.926672    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:26:10.926819 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:10.926726    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" event={"ID":"590eb23d-6765-47e8-8bcd-9e96e06e3e55","Type":"ContainerStarted","Data":"c48c8bbd35c45dd384b04ff63ddbb7608c185850a7899efa4592865f131e2c79"}
Sep 17 19:26:10.928366 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:10.928342    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" event={"ID":"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3","Type":"ContainerStarted","Data":"5c63e32e84f6d2b44c430226791af7074457f9b4f8da510ce1b35529ea8866e1"}
Sep 17 19:26:10.928520 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:10.928393    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" event={"ID":"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3","Type":"ContainerStarted","Data":"254d13f1353c4d068f50610f665a8e925860380b4fd977f316ea0b6902514186"}
Sep 17 19:26:10.928520 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:10.928422    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" event={"ID":"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3","Type":"ContainerStarted","Data":"fc9a1af7ac3068626b50b2af6bca212e94a559877c3ee8eafafd8d8e7cdbfa4c"}
Sep 17 19:26:10.928520 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:10.928487    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:26:10.949254 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:10.949198    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" podStartSLOduration=1.949182409 podStartE2EDuration="1.949182409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:26:10.947154737 +0000 UTC m=+1174.775012970" watchObservedRunningTime="2026-09-17 19:26:10.949182409 +0000 UTC m=+1174.777040642"
Sep 17 19:26:10.968406 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:10.968294    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" podStartSLOduration=1.968277358 podStartE2EDuration="1.968277358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:26:10.967160885 +0000 UTC m=+1174.795019161" watchObservedRunningTime="2026-09-17 19:26:10.968277358 +0000 UTC m=+1174.796135592"
Sep 17 19:26:11.297617 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:11.297574    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 19:26:11.301935 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:11.301905    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 19:26:11.933659 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:11.933625    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:26:11.934098 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:11.933681    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:26:11.934752 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:11.934728    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:26:11.934752 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:11.934742    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 19:26:12.938383 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:12.938345    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:26:12.938866 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:12.938428    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 19:26:13.310239 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.310211    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:26:13.394298 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.394261    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3743c8e9-c568-4a36-9b7c-b4d1653ff448-proxy-tls\") pod \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\" (UID: \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\") "
Sep 17 19:26:13.394298 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.394301    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3743c8e9-c568-4a36-9b7c-b4d1653ff448-kube-rbac-proxy-sar-config\") pod \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\" (UID: \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\") "
Sep 17 19:26:13.394646 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.394322    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5df7\" (UniqueName: \"kubernetes.io/projected/3743c8e9-c568-4a36-9b7c-b4d1653ff448-kube-api-access-s5df7\") pod \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\" (UID: \"3743c8e9-c568-4a36-9b7c-b4d1653ff448\") "
Sep 17 19:26:13.394778 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.394754    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3743c8e9-c568-4a36-9b7c-b4d1653ff448-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3743c8e9-c568-4a36-9b7c-b4d1653ff448" (UID: "3743c8e9-c568-4a36-9b7c-b4d1653ff448"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:26:13.396425 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.396402    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3743c8e9-c568-4a36-9b7c-b4d1653ff448-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3743c8e9-c568-4a36-9b7c-b4d1653ff448" (UID: "3743c8e9-c568-4a36-9b7c-b4d1653ff448"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:26:13.396532 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.396403    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3743c8e9-c568-4a36-9b7c-b4d1653ff448-kube-api-access-s5df7" (OuterVolumeSpecName: "kube-api-access-s5df7") pod "3743c8e9-c568-4a36-9b7c-b4d1653ff448" (UID: "3743c8e9-c568-4a36-9b7c-b4d1653ff448"). InnerVolumeSpecName "kube-api-access-s5df7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:26:13.495540 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.495431    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3743c8e9-c568-4a36-9b7c-b4d1653ff448-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:26:13.495540 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.495479    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3743c8e9-c568-4a36-9b7c-b4d1653ff448-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:26:13.495540 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.495494    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5df7\" (UniqueName: \"kubernetes.io/projected/3743c8e9-c568-4a36-9b7c-b4d1653ff448-kube-api-access-s5df7\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:26:13.633100 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.633075    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:26:13.697295 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.697253    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-kube-rbac-proxy-sar-config\") pod \"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8\" (UID: \"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8\") "
Sep 17 19:26:13.697475 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.697308    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-proxy-tls\") pod \"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8\" (UID: \"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8\") "
Sep 17 19:26:13.697475 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.697332    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksxmx\" (UniqueName: \"kubernetes.io/projected/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-kube-api-access-ksxmx\") pod \"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8\" (UID: \"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8\") "
Sep 17 19:26:13.697664 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.697641    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" (UID: "d17cd3d4-707c-4fe7-8f28-7f8278f0abc8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:26:13.699388 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.699367    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" (UID: "d17cd3d4-707c-4fe7-8f28-7f8278f0abc8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:26:13.699481 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.699425    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-kube-api-access-ksxmx" (OuterVolumeSpecName: "kube-api-access-ksxmx") pod "d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" (UID: "d17cd3d4-707c-4fe7-8f28-7f8278f0abc8"). InnerVolumeSpecName "kube-api-access-ksxmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:26:13.797947 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.797909    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:26:13.797947 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.797944    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:26:13.797947 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.797955    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksxmx\" (UniqueName: \"kubernetes.io/projected/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8-kube-api-access-ksxmx\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:26:13.945865 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.945832    2583 generic.go:356] "Generic (PLEG): container finished" podID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerID="6ade7e702b0f33ade11d645bc7b5c4cc46024da3cc68fdf0d33246d20884eb73" exitCode=0
Sep 17 19:26:13.946301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.945902    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"
Sep 17 19:26:13.946301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.945919    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" event={"ID":"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8","Type":"ContainerDied","Data":"6ade7e702b0f33ade11d645bc7b5c4cc46024da3cc68fdf0d33246d20884eb73"}
Sep 17 19:26:13.946301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.946007    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22" event={"ID":"d17cd3d4-707c-4fe7-8f28-7f8278f0abc8","Type":"ContainerDied","Data":"d786a5f69abe1b126dea6da567acc7a280adeabf322dc86ce19689cc8f1ab05c"}
Sep 17 19:26:13.946301 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.946038    2583 scope.go:117] "RemoveContainer" containerID="2cd521e83fe30096ec72e4937b892152bf3c771e7771b72a57084bfd61501efe"
Sep 17 19:26:13.947694 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.947668    2583 generic.go:356] "Generic (PLEG): container finished" podID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerID="ba8a7e1f75a241e9790a69e270ec0e24a9f3f410dc47f81ef01c7d2328f42571" exitCode=0
Sep 17 19:26:13.947832 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.947696    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" event={"ID":"3743c8e9-c568-4a36-9b7c-b4d1653ff448","Type":"ContainerDied","Data":"ba8a7e1f75a241e9790a69e270ec0e24a9f3f410dc47f81ef01c7d2328f42571"}
Sep 17 19:26:13.947832 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.947740    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh" event={"ID":"3743c8e9-c568-4a36-9b7c-b4d1653ff448","Type":"ContainerDied","Data":"aa8d8da0beacf6fc846cd8aed872a53dd92fa9ee51e2478df1bc99e068382eeb"}
Sep 17 19:26:13.947832 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.947747    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"
Sep 17 19:26:13.954676 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.954639    2583 scope.go:117] "RemoveContainer" containerID="6ade7e702b0f33ade11d645bc7b5c4cc46024da3cc68fdf0d33246d20884eb73"
Sep 17 19:26:13.962537 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.962516    2583 scope.go:117] "RemoveContainer" containerID="2cd521e83fe30096ec72e4937b892152bf3c771e7771b72a57084bfd61501efe"
Sep 17 19:26:13.962829 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:26:13.962812    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd521e83fe30096ec72e4937b892152bf3c771e7771b72a57084bfd61501efe\": container with ID starting with 2cd521e83fe30096ec72e4937b892152bf3c771e7771b72a57084bfd61501efe not found: ID does not exist" containerID="2cd521e83fe30096ec72e4937b892152bf3c771e7771b72a57084bfd61501efe"
Sep 17 19:26:13.962880 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.962835    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd521e83fe30096ec72e4937b892152bf3c771e7771b72a57084bfd61501efe"} err="failed to get container status \"2cd521e83fe30096ec72e4937b892152bf3c771e7771b72a57084bfd61501efe\": rpc error: code = NotFound desc = could not find container \"2cd521e83fe30096ec72e4937b892152bf3c771e7771b72a57084bfd61501efe\": container with ID starting with 2cd521e83fe30096ec72e4937b892152bf3c771e7771b72a57084bfd61501efe not found: ID does not exist"
Sep 17 19:26:13.962880 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.962851    2583 scope.go:117] "RemoveContainer" containerID="6ade7e702b0f33ade11d645bc7b5c4cc46024da3cc68fdf0d33246d20884eb73"
Sep 17 19:26:13.963064 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:26:13.963051    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ade7e702b0f33ade11d645bc7b5c4cc46024da3cc68fdf0d33246d20884eb73\": container with ID starting with 6ade7e702b0f33ade11d645bc7b5c4cc46024da3cc68fdf0d33246d20884eb73 not found: ID does not exist" containerID="6ade7e702b0f33ade11d645bc7b5c4cc46024da3cc68fdf0d33246d20884eb73"
Sep 17 19:26:13.963100 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.963067    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ade7e702b0f33ade11d645bc7b5c4cc46024da3cc68fdf0d33246d20884eb73"} err="failed to get container status \"6ade7e702b0f33ade11d645bc7b5c4cc46024da3cc68fdf0d33246d20884eb73\": rpc error: code = NotFound desc = could not find container \"6ade7e702b0f33ade11d645bc7b5c4cc46024da3cc68fdf0d33246d20884eb73\": container with ID starting with 6ade7e702b0f33ade11d645bc7b5c4cc46024da3cc68fdf0d33246d20884eb73 not found: ID does not exist"
Sep 17 19:26:13.963100 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.963078    2583 scope.go:117] "RemoveContainer" containerID="3a67000eedf150ac33674843d87dce5fef14cf4c84fc15483046c4c0561d84c5"
Sep 17 19:26:13.970856 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.970838    2583 scope.go:117] "RemoveContainer" containerID="ba8a7e1f75a241e9790a69e270ec0e24a9f3f410dc47f81ef01c7d2328f42571"
Sep 17 19:26:13.972131 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.972096    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"]
Sep 17 19:26:13.974641 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.974620    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c825-predictor-67999bcd6c-r4ljh"]
Sep 17 19:26:13.979334 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.979316    2583 scope.go:117] "RemoveContainer" containerID="3a67000eedf150ac33674843d87dce5fef14cf4c84fc15483046c4c0561d84c5"
Sep 17 19:26:13.979695 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:26:13.979675    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a67000eedf150ac33674843d87dce5fef14cf4c84fc15483046c4c0561d84c5\": container with ID starting with 3a67000eedf150ac33674843d87dce5fef14cf4c84fc15483046c4c0561d84c5 not found: ID does not exist" containerID="3a67000eedf150ac33674843d87dce5fef14cf4c84fc15483046c4c0561d84c5"
Sep 17 19:26:13.979764 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.979706    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a67000eedf150ac33674843d87dce5fef14cf4c84fc15483046c4c0561d84c5"} err="failed to get container status \"3a67000eedf150ac33674843d87dce5fef14cf4c84fc15483046c4c0561d84c5\": rpc error: code = NotFound desc = could not find container \"3a67000eedf150ac33674843d87dce5fef14cf4c84fc15483046c4c0561d84c5\": container with ID starting with 3a67000eedf150ac33674843d87dce5fef14cf4c84fc15483046c4c0561d84c5 not found: ID does not exist"
Sep 17 19:26:13.979764 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.979730    2583 scope.go:117] "RemoveContainer" containerID="ba8a7e1f75a241e9790a69e270ec0e24a9f3f410dc47f81ef01c7d2328f42571"
Sep 17 19:26:13.979987 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:26:13.979970    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8a7e1f75a241e9790a69e270ec0e24a9f3f410dc47f81ef01c7d2328f42571\": container with ID starting with ba8a7e1f75a241e9790a69e270ec0e24a9f3f410dc47f81ef01c7d2328f42571 not found: ID does not exist" containerID="ba8a7e1f75a241e9790a69e270ec0e24a9f3f410dc47f81ef01c7d2328f42571"
Sep 17 19:26:13.980028 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.979995    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8a7e1f75a241e9790a69e270ec0e24a9f3f410dc47f81ef01c7d2328f42571"} err="failed to get container status \"ba8a7e1f75a241e9790a69e270ec0e24a9f3f410dc47f81ef01c7d2328f42571\": rpc error: code = NotFound desc = could not find container \"ba8a7e1f75a241e9790a69e270ec0e24a9f3f410dc47f81ef01c7d2328f42571\": container with ID starting with ba8a7e1f75a241e9790a69e270ec0e24a9f3f410dc47f81ef01c7d2328f42571 not found: ID does not exist"
Sep 17 19:26:13.984296 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.984278    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"]
Sep 17 19:26:13.989957 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:13.989939    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8c825-predictor-7f9d986954-pxj22"]
Sep 17 19:26:14.456805 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:14.453836    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" path="/var/lib/kubelet/pods/3743c8e9-c568-4a36-9b7c-b4d1653ff448/volumes"
Sep 17 19:26:14.456805 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:14.454507    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" path="/var/lib/kubelet/pods/d17cd3d4-707c-4fe7-8f28-7f8278f0abc8/volumes"
Sep 17 19:26:17.943248 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:17.943216    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:26:17.943822 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:17.943317    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:26:17.943822 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:17.943764    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 19:26:17.943822 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:17.943770    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:26:27.943797 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:27.943747    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:26:27.943797 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:27.943771    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 19:26:37.944166 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:37.944123    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 19:26:37.944561 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:37.944123    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:26:38.420633 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:38.420603    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:26:38.422967 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:38.422942    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:26:47.943775 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:47.943732    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 19:26:47.944204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:47.943737    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:26:57.943994 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:57.943952    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 19:26:57.944490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:26:57.943956    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 19:27:07.944780 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:07.944744    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:27:07.945227 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:07.945210    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:27:10.563155 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.563111    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"]
Sep 17 19:27:10.563735 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.563416    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kserve-container" containerID="cri-o://963cf20f90f326a14b23d12e3fdab58c989e20927a891939b0237cda59782937" gracePeriod=30
Sep 17 19:27:10.563735 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.563484    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kube-rbac-proxy" containerID="cri-o://ec4e52e57f1f9249672f54d5e4a3c1415d566065fb01db39ebbc5440fa356d10" gracePeriod=30
Sep 17 19:27:10.624040 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.624003    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"]
Sep 17 19:27:10.624264 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.624228    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kserve-container" containerID="cri-o://0a210b7928ebb2b38958150f89480f242be94ee0d75b8a7a2458befd567bde7e" gracePeriod=30
Sep 17 19:27:10.624396 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.624285    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kube-rbac-proxy" containerID="cri-o://62a28ead310dc6418def74e41a178aa22b836e81dfd6f8db51a3461227cf9764" gracePeriod=30
Sep 17 19:27:10.720668 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.720634    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"]
Sep 17 19:27:10.721038 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.721026    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kube-rbac-proxy"
Sep 17 19:27:10.721081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.721040    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kube-rbac-proxy"
Sep 17 19:27:10.721081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.721058    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerName="kube-rbac-proxy"
Sep 17 19:27:10.721081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.721066    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerName="kube-rbac-proxy"
Sep 17 19:27:10.721081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.721074    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kserve-container"
Sep 17 19:27:10.721081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.721079    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kserve-container"
Sep 17 19:27:10.721233 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.721098    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerName="kserve-container"
Sep 17 19:27:10.721233 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.721105    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerName="kserve-container"
Sep 17 19:27:10.721233 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.721166    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerName="kube-rbac-proxy"
Sep 17 19:27:10.721233 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.721180    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kserve-container"
Sep 17 19:27:10.721233 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.721188    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d17cd3d4-707c-4fe7-8f28-7f8278f0abc8" containerName="kube-rbac-proxy"
Sep 17 19:27:10.721233 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.721193    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3743c8e9-c568-4a36-9b7c-b4d1653ff448" containerName="kserve-container"
Sep 17 19:27:10.724329 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.724312    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:27:10.727652 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.727629    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-30eba-predictor-serving-cert\""
Sep 17 19:27:10.727788 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.727629    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-30eba-kube-rbac-proxy-sar-config\""
Sep 17 19:27:10.737705 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.737685    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"]
Sep 17 19:27:10.799840 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.799806    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktkxq\" (UniqueName: \"kubernetes.io/projected/14e323e0-7081-47e9-9bc8-fae817f75bbe-kube-api-access-ktkxq\") pod \"success-200-isvc-30eba-predictor-68c87bd657-pdbdn\" (UID: \"14e323e0-7081-47e9-9bc8-fae817f75bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:27:10.800025 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.799854    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14e323e0-7081-47e9-9bc8-fae817f75bbe-proxy-tls\") pod \"success-200-isvc-30eba-predictor-68c87bd657-pdbdn\" (UID: \"14e323e0-7081-47e9-9bc8-fae817f75bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:27:10.800025 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.799877    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14e323e0-7081-47e9-9bc8-fae817f75bbe-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-30eba-predictor-68c87bd657-pdbdn\" (UID: \"14e323e0-7081-47e9-9bc8-fae817f75bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:27:10.869011 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.868923    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"]
Sep 17 19:27:10.872475 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.872434    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:27:10.875231 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.875204    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-30eba-predictor-serving-cert\""
Sep 17 19:27:10.876229 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.876182    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-30eba-kube-rbac-proxy-sar-config\""
Sep 17 19:27:10.885149 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.885122    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"]
Sep 17 19:27:10.900824 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.900788    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktkxq\" (UniqueName: \"kubernetes.io/projected/14e323e0-7081-47e9-9bc8-fae817f75bbe-kube-api-access-ktkxq\") pod \"success-200-isvc-30eba-predictor-68c87bd657-pdbdn\" (UID: \"14e323e0-7081-47e9-9bc8-fae817f75bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:27:10.901014 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.900830    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/631d021b-4a9a-4b39-80a6-54cf8b103db4-proxy-tls\") pod \"error-404-isvc-30eba-predictor-8cb6986c6-fk824\" (UID: \"631d021b-4a9a-4b39-80a6-54cf8b103db4\") " pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:27:10.901014 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.900869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/631d021b-4a9a-4b39-80a6-54cf8b103db4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-30eba-predictor-8cb6986c6-fk824\" (UID: \"631d021b-4a9a-4b39-80a6-54cf8b103db4\") " pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:27:10.901014 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.900896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14e323e0-7081-47e9-9bc8-fae817f75bbe-proxy-tls\") pod \"success-200-isvc-30eba-predictor-68c87bd657-pdbdn\" (UID: \"14e323e0-7081-47e9-9bc8-fae817f75bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:27:10.901014 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.900925    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14e323e0-7081-47e9-9bc8-fae817f75bbe-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-30eba-predictor-68c87bd657-pdbdn\" (UID: \"14e323e0-7081-47e9-9bc8-fae817f75bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:27:10.901014 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.901000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcptg\" (UniqueName: \"kubernetes.io/projected/631d021b-4a9a-4b39-80a6-54cf8b103db4-kube-api-access-tcptg\") pod \"error-404-isvc-30eba-predictor-8cb6986c6-fk824\" (UID: \"631d021b-4a9a-4b39-80a6-54cf8b103db4\") " pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:27:10.901606 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.901589    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14e323e0-7081-47e9-9bc8-fae817f75bbe-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-30eba-predictor-68c87bd657-pdbdn\" (UID: \"14e323e0-7081-47e9-9bc8-fae817f75bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:27:10.903778 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.903752    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14e323e0-7081-47e9-9bc8-fae817f75bbe-proxy-tls\") pod \"success-200-isvc-30eba-predictor-68c87bd657-pdbdn\" (UID: \"14e323e0-7081-47e9-9bc8-fae817f75bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:27:10.912587 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:10.912564    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktkxq\" (UniqueName: \"kubernetes.io/projected/14e323e0-7081-47e9-9bc8-fae817f75bbe-kube-api-access-ktkxq\") pod \"success-200-isvc-30eba-predictor-68c87bd657-pdbdn\" (UID: \"14e323e0-7081-47e9-9bc8-fae817f75bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:27:11.001674 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:11.001618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/631d021b-4a9a-4b39-80a6-54cf8b103db4-proxy-tls\") pod \"error-404-isvc-30eba-predictor-8cb6986c6-fk824\" (UID: \"631d021b-4a9a-4b39-80a6-54cf8b103db4\") " pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:27:11.001674 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:11.001683    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/631d021b-4a9a-4b39-80a6-54cf8b103db4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-30eba-predictor-8cb6986c6-fk824\" (UID: \"631d021b-4a9a-4b39-80a6-54cf8b103db4\") " pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:27:11.001923 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:11.001741    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcptg\" (UniqueName: \"kubernetes.io/projected/631d021b-4a9a-4b39-80a6-54cf8b103db4-kube-api-access-tcptg\") pod \"error-404-isvc-30eba-predictor-8cb6986c6-fk824\" (UID: \"631d021b-4a9a-4b39-80a6-54cf8b103db4\") " pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:27:11.001923 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:27:11.001768    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-30eba-predictor-serving-cert: secret "error-404-isvc-30eba-predictor-serving-cert" not found
Sep 17 19:27:11.001923 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:27:11.001830    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/631d021b-4a9a-4b39-80a6-54cf8b103db4-proxy-tls podName:631d021b-4a9a-4b39-80a6-54cf8b103db4 nodeName:}" failed. No retries permitted until 2026-09-17 19:27:11.501812943 +0000 UTC m=+1235.329671160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/631d021b-4a9a-4b39-80a6-54cf8b103db4-proxy-tls") pod "error-404-isvc-30eba-predictor-8cb6986c6-fk824" (UID: "631d021b-4a9a-4b39-80a6-54cf8b103db4") : secret "error-404-isvc-30eba-predictor-serving-cert" not found
Sep 17 19:27:11.002388 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:11.002370    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/631d021b-4a9a-4b39-80a6-54cf8b103db4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-30eba-predictor-8cb6986c6-fk824\" (UID: \"631d021b-4a9a-4b39-80a6-54cf8b103db4\") " pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:27:11.010495 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:11.010469    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcptg\" (UniqueName: \"kubernetes.io/projected/631d021b-4a9a-4b39-80a6-54cf8b103db4-kube-api-access-tcptg\") pod \"error-404-isvc-30eba-predictor-8cb6986c6-fk824\" (UID: \"631d021b-4a9a-4b39-80a6-54cf8b103db4\") " pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:27:11.034601 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:11.034565    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:27:11.220792 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:11.220695    2583 generic.go:356] "Generic (PLEG): container finished" podID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerID="62a28ead310dc6418def74e41a178aa22b836e81dfd6f8db51a3461227cf9764" exitCode=2
Sep 17 19:27:11.220792 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:11.220779    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" event={"ID":"3bc44fcf-6903-4ce1-988d-2d66fe1da04d","Type":"ContainerDied","Data":"62a28ead310dc6418def74e41a178aa22b836e81dfd6f8db51a3461227cf9764"}
Sep 17 19:27:11.222743 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:11.222716    2583 generic.go:356] "Generic (PLEG): container finished" podID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerID="ec4e52e57f1f9249672f54d5e4a3c1415d566065fb01db39ebbc5440fa356d10" exitCode=2
Sep 17 19:27:11.222869 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:11.222776    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" event={"ID":"600dbcf2-78cf-4eb9-b8b5-95ff97b91539","Type":"ContainerDied","Data":"ec4e52e57f1f9249672f54d5e4a3c1415d566065fb01db39ebbc5440fa356d10"}
Sep 17 19:27:11.308481 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:11.308363    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"]
Sep 17 19:27:11.310665 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:27:11.310641    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e323e0_7081_47e9_9bc8_fae817f75bbe.slice/crio-66028b5733f95409c925f7fc414743e9c6ecf1d7723d29d0d2efde1f9217b795 WatchSource:0}: Error finding container 66028b5733f95409c925f7fc414743e9c6ecf1d7723d29d0d2efde1f9217b795: Status 404 returned error can't find the container with id 66028b5733f95409c925f7fc414743e9c6ecf1d7723d29d0d2efde1f9217b795
Sep 17 19:27:11.506414 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:11.506303    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/631d021b-4a9a-4b39-80a6-54cf8b103db4-proxy-tls\") pod \"error-404-isvc-30eba-predictor-8cb6986c6-fk824\" (UID: \"631d021b-4a9a-4b39-80a6-54cf8b103db4\") " pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:27:11.508727 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:11.508695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/631d021b-4a9a-4b39-80a6-54cf8b103db4-proxy-tls\") pod \"error-404-isvc-30eba-predictor-8cb6986c6-fk824\" (UID: \"631d021b-4a9a-4b39-80a6-54cf8b103db4\") " pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:27:11.782924 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:11.782825    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:27:12.054372 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.054215    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"]
Sep 17 19:27:12.057012 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:27:12.056980    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631d021b_4a9a_4b39_80a6_54cf8b103db4.slice/crio-08fdf15706bb365cf1b2bc2eb1f8c43ac6251bedff375b0910666486ed65ee98 WatchSource:0}: Error finding container 08fdf15706bb365cf1b2bc2eb1f8c43ac6251bedff375b0910666486ed65ee98: Status 404 returned error can't find the container with id 08fdf15706bb365cf1b2bc2eb1f8c43ac6251bedff375b0910666486ed65ee98
Sep 17 19:27:12.229809 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.229763    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" event={"ID":"631d021b-4a9a-4b39-80a6-54cf8b103db4","Type":"ContainerStarted","Data":"71018ad550c583828721c83d47af13220eeaf565c7758e5fbd77994ec382f6e7"}
Sep 17 19:27:12.230117 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.229870    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:27:12.230117 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.229901    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:27:12.230117 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.229927    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" event={"ID":"631d021b-4a9a-4b39-80a6-54cf8b103db4","Type":"ContainerStarted","Data":"21cede2f8c7a03ae55dc9b017dda43328647f67d4a3ec88c1b1eff0b6ebdc056"}
Sep 17 19:27:12.230117 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.229953    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" event={"ID":"631d021b-4a9a-4b39-80a6-54cf8b103db4","Type":"ContainerStarted","Data":"08fdf15706bb365cf1b2bc2eb1f8c43ac6251bedff375b0910666486ed65ee98"}
Sep 17 19:27:12.234276 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.234235    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 19:27:12.234889 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.234867    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" event={"ID":"14e323e0-7081-47e9-9bc8-fae817f75bbe","Type":"ContainerStarted","Data":"4d4d27606b15460dc92908384998319243ccfb65adc7e156f2d332ff44af977b"}
Sep 17 19:27:12.234983 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.234927    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" event={"ID":"14e323e0-7081-47e9-9bc8-fae817f75bbe","Type":"ContainerStarted","Data":"85bd81ef19c81d63294889c8f1633e559524878974d59ec66381f829e06872af"}
Sep 17 19:27:12.234983 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.234951    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" event={"ID":"14e323e0-7081-47e9-9bc8-fae817f75bbe","Type":"ContainerStarted","Data":"66028b5733f95409c925f7fc414743e9c6ecf1d7723d29d0d2efde1f9217b795"}
Sep 17 19:27:12.235061 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.234990    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:27:12.235061 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.235042    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:27:12.236109 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.236085    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 19:27:12.265881 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.265831    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" podStartSLOduration=2.265812682 podStartE2EDuration="2.265812682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:27:12.264911355 +0000 UTC m=+1236.092769588" watchObservedRunningTime="2026-09-17 19:27:12.265812682 +0000 UTC m=+1236.093670918"
Sep 17 19:27:12.304203 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.304152    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" podStartSLOduration=2.3041370199999998 podStartE2EDuration="2.30413702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:27:12.302221714 +0000 UTC m=+1236.130079947" watchObservedRunningTime="2026-09-17 19:27:12.30413702 +0000 UTC m=+1236.131995253"
Sep 17 19:27:12.601806 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.601764    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 19:27:12.608231 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:12.608201    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 19:27:13.239996 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:13.239958    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 19:27:13.240481 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:13.240038    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 19:27:13.606482 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:13.606411    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 19:27:13.612494 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:13.612461    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 19:27:14.830660 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:14.830633    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:27:14.854856 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:14.854829    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:27:14.938630 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:14.938521    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-kube-rbac-proxy-sar-config\") pod \"600dbcf2-78cf-4eb9-b8b5-95ff97b91539\" (UID: \"600dbcf2-78cf-4eb9-b8b5-95ff97b91539\") "
Sep 17 19:27:14.938630 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:14.938606    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-proxy-tls\") pod \"3bc44fcf-6903-4ce1-988d-2d66fe1da04d\" (UID: \"3bc44fcf-6903-4ce1-988d-2d66fe1da04d\") "
Sep 17 19:27:14.938863 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:14.938644    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-kube-rbac-proxy-sar-config\") pod \"3bc44fcf-6903-4ce1-988d-2d66fe1da04d\" (UID: \"3bc44fcf-6903-4ce1-988d-2d66fe1da04d\") "
Sep 17 19:27:14.938863 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:14.938670    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm79t\" (UniqueName: \"kubernetes.io/projected/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-kube-api-access-qm79t\") pod \"600dbcf2-78cf-4eb9-b8b5-95ff97b91539\" (UID: \"600dbcf2-78cf-4eb9-b8b5-95ff97b91539\") "
Sep 17 19:27:14.938863 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:14.938695    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctk6r\" (UniqueName: \"kubernetes.io/projected/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-kube-api-access-ctk6r\") pod \"3bc44fcf-6903-4ce1-988d-2d66fe1da04d\" (UID: \"3bc44fcf-6903-4ce1-988d-2d66fe1da04d\") "
Sep 17 19:27:14.938863 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:14.938749    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-proxy-tls\") pod \"600dbcf2-78cf-4eb9-b8b5-95ff97b91539\" (UID: \"600dbcf2-78cf-4eb9-b8b5-95ff97b91539\") "
Sep 17 19:27:14.939077 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:14.938945    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "600dbcf2-78cf-4eb9-b8b5-95ff97b91539" (UID: "600dbcf2-78cf-4eb9-b8b5-95ff97b91539"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:27:14.939077 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:14.938977    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3bc44fcf-6903-4ce1-988d-2d66fe1da04d" (UID: "3bc44fcf-6903-4ce1-988d-2d66fe1da04d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:27:14.940942 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:14.940900    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "600dbcf2-78cf-4eb9-b8b5-95ff97b91539" (UID: "600dbcf2-78cf-4eb9-b8b5-95ff97b91539"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:14.940942 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:14.940908    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-kube-api-access-ctk6r" (OuterVolumeSpecName: "kube-api-access-ctk6r") pod "3bc44fcf-6903-4ce1-988d-2d66fe1da04d" (UID: "3bc44fcf-6903-4ce1-988d-2d66fe1da04d"). InnerVolumeSpecName "kube-api-access-ctk6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:14.941091 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:14.940952    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-kube-api-access-qm79t" (OuterVolumeSpecName: "kube-api-access-qm79t") pod "600dbcf2-78cf-4eb9-b8b5-95ff97b91539" (UID: "600dbcf2-78cf-4eb9-b8b5-95ff97b91539"). InnerVolumeSpecName "kube-api-access-qm79t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:14.941091 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:14.940976    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3bc44fcf-6903-4ce1-988d-2d66fe1da04d" (UID: "3bc44fcf-6903-4ce1-988d-2d66fe1da04d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:15.040303 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.040261    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:27:15.040303 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.040294    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:27:15.040303 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.040305    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:27:15.040303 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.040313    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:27:15.040609 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.040321    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm79t\" (UniqueName: \"kubernetes.io/projected/600dbcf2-78cf-4eb9-b8b5-95ff97b91539-kube-api-access-qm79t\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:27:15.040609 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.040330    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctk6r\" (UniqueName: \"kubernetes.io/projected/3bc44fcf-6903-4ce1-988d-2d66fe1da04d-kube-api-access-ctk6r\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:27:15.250407 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.250308    2583 generic.go:356] "Generic (PLEG): container finished" podID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerID="0a210b7928ebb2b38958150f89480f242be94ee0d75b8a7a2458befd567bde7e" exitCode=0
Sep 17 19:27:15.250407 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.250358    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"
Sep 17 19:27:15.250407 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.250365    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" event={"ID":"3bc44fcf-6903-4ce1-988d-2d66fe1da04d","Type":"ContainerDied","Data":"0a210b7928ebb2b38958150f89480f242be94ee0d75b8a7a2458befd567bde7e"}
Sep 17 19:27:15.250706 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.250419    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs" event={"ID":"3bc44fcf-6903-4ce1-988d-2d66fe1da04d","Type":"ContainerDied","Data":"3cbfe3e9a4407f82db8d59e18247034d0f9fea6192dfec7ec0df7139a25ed055"}
Sep 17 19:27:15.250706 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.250470    2583 scope.go:117] "RemoveContainer" containerID="62a28ead310dc6418def74e41a178aa22b836e81dfd6f8db51a3461227cf9764"
Sep 17 19:27:15.252240 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.252204    2583 generic.go:356] "Generic (PLEG): container finished" podID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerID="963cf20f90f326a14b23d12e3fdab58c989e20927a891939b0237cda59782937" exitCode=0
Sep 17 19:27:15.252347 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.252256    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"
Sep 17 19:27:15.252347 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.252279    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" event={"ID":"600dbcf2-78cf-4eb9-b8b5-95ff97b91539","Type":"ContainerDied","Data":"963cf20f90f326a14b23d12e3fdab58c989e20927a891939b0237cda59782937"}
Sep 17 19:27:15.252347 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.252328    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh" event={"ID":"600dbcf2-78cf-4eb9-b8b5-95ff97b91539","Type":"ContainerDied","Data":"66eb5a3ee8f90a595d25ae3c7f6ef854fab727482a22baa6e9c284647ad2047a"}
Sep 17 19:27:15.259005 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.258983    2583 scope.go:117] "RemoveContainer" containerID="0a210b7928ebb2b38958150f89480f242be94ee0d75b8a7a2458befd567bde7e"
Sep 17 19:27:15.267165 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.267149    2583 scope.go:117] "RemoveContainer" containerID="62a28ead310dc6418def74e41a178aa22b836e81dfd6f8db51a3461227cf9764"
Sep 17 19:27:15.267486 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:27:15.267468    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a28ead310dc6418def74e41a178aa22b836e81dfd6f8db51a3461227cf9764\": container with ID starting with 62a28ead310dc6418def74e41a178aa22b836e81dfd6f8db51a3461227cf9764 not found: ID does not exist" containerID="62a28ead310dc6418def74e41a178aa22b836e81dfd6f8db51a3461227cf9764"
Sep 17 19:27:15.267539 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.267496    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a28ead310dc6418def74e41a178aa22b836e81dfd6f8db51a3461227cf9764"} err="failed to get container status \"62a28ead310dc6418def74e41a178aa22b836e81dfd6f8db51a3461227cf9764\": rpc error: code = NotFound desc = could not find container \"62a28ead310dc6418def74e41a178aa22b836e81dfd6f8db51a3461227cf9764\": container with ID starting with 62a28ead310dc6418def74e41a178aa22b836e81dfd6f8db51a3461227cf9764 not found: ID does not exist"
Sep 17 19:27:15.267539 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.267513    2583 scope.go:117] "RemoveContainer" containerID="0a210b7928ebb2b38958150f89480f242be94ee0d75b8a7a2458befd567bde7e"
Sep 17 19:27:15.267773 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:27:15.267757    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a210b7928ebb2b38958150f89480f242be94ee0d75b8a7a2458befd567bde7e\": container with ID starting with 0a210b7928ebb2b38958150f89480f242be94ee0d75b8a7a2458befd567bde7e not found: ID does not exist" containerID="0a210b7928ebb2b38958150f89480f242be94ee0d75b8a7a2458befd567bde7e"
Sep 17 19:27:15.267816 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.267782    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a210b7928ebb2b38958150f89480f242be94ee0d75b8a7a2458befd567bde7e"} err="failed to get container status \"0a210b7928ebb2b38958150f89480f242be94ee0d75b8a7a2458befd567bde7e\": rpc error: code = NotFound desc = could not find container \"0a210b7928ebb2b38958150f89480f242be94ee0d75b8a7a2458befd567bde7e\": container with ID starting with 0a210b7928ebb2b38958150f89480f242be94ee0d75b8a7a2458befd567bde7e not found: ID does not exist"
Sep 17 19:27:15.267816 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.267797    2583 scope.go:117] "RemoveContainer" containerID="ec4e52e57f1f9249672f54d5e4a3c1415d566065fb01db39ebbc5440fa356d10"
Sep 17 19:27:15.275354 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.275330    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"]
Sep 17 19:27:15.275557 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.275528    2583 scope.go:117] "RemoveContainer" containerID="963cf20f90f326a14b23d12e3fdab58c989e20927a891939b0237cda59782937"
Sep 17 19:27:15.283390 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.283368    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fcbf3-predictor-6cdb6c88c8-82pqs"]
Sep 17 19:27:15.283464 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.283409    2583 scope.go:117] "RemoveContainer" containerID="ec4e52e57f1f9249672f54d5e4a3c1415d566065fb01db39ebbc5440fa356d10"
Sep 17 19:27:15.283761 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:27:15.283745    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec4e52e57f1f9249672f54d5e4a3c1415d566065fb01db39ebbc5440fa356d10\": container with ID starting with ec4e52e57f1f9249672f54d5e4a3c1415d566065fb01db39ebbc5440fa356d10 not found: ID does not exist" containerID="ec4e52e57f1f9249672f54d5e4a3c1415d566065fb01db39ebbc5440fa356d10"
Sep 17 19:27:15.283809 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.283768    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec4e52e57f1f9249672f54d5e4a3c1415d566065fb01db39ebbc5440fa356d10"} err="failed to get container status \"ec4e52e57f1f9249672f54d5e4a3c1415d566065fb01db39ebbc5440fa356d10\": rpc error: code = NotFound desc = could not find container \"ec4e52e57f1f9249672f54d5e4a3c1415d566065fb01db39ebbc5440fa356d10\": container with ID starting with ec4e52e57f1f9249672f54d5e4a3c1415d566065fb01db39ebbc5440fa356d10 not found: ID does not exist"
Sep 17 19:27:15.283809 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.283787    2583 scope.go:117] "RemoveContainer" containerID="963cf20f90f326a14b23d12e3fdab58c989e20927a891939b0237cda59782937"
Sep 17 19:27:15.284025 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:27:15.284010    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963cf20f90f326a14b23d12e3fdab58c989e20927a891939b0237cda59782937\": container with ID starting with 963cf20f90f326a14b23d12e3fdab58c989e20927a891939b0237cda59782937 not found: ID does not exist" containerID="963cf20f90f326a14b23d12e3fdab58c989e20927a891939b0237cda59782937"
Sep 17 19:27:15.284068 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.284030    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963cf20f90f326a14b23d12e3fdab58c989e20927a891939b0237cda59782937"} err="failed to get container status \"963cf20f90f326a14b23d12e3fdab58c989e20927a891939b0237cda59782937\": rpc error: code = NotFound desc = could not find container \"963cf20f90f326a14b23d12e3fdab58c989e20927a891939b0237cda59782937\": container with ID starting with 963cf20f90f326a14b23d12e3fdab58c989e20927a891939b0237cda59782937 not found: ID does not exist"
Sep 17 19:27:15.295892 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.295855    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"]
Sep 17 19:27:15.299955 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:15.299928    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fcbf3-predictor-76db4cd48c-zthzh"]
Sep 17 19:27:16.452838 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:16.452804    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" path="/var/lib/kubelet/pods/3bc44fcf-6903-4ce1-988d-2d66fe1da04d/volumes"
Sep 17 19:27:16.453353 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:16.453335    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" path="/var/lib/kubelet/pods/600dbcf2-78cf-4eb9-b8b5-95ff97b91539/volumes"
Sep 17 19:27:18.246338 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:18.246306    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:27:18.248308 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:18.248271    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 19:27:18.250798 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:18.250777    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:27:18.251154 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:18.251133    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 19:27:28.248221 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:28.248180    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 19:27:28.251556 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:28.251526    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 19:27:30.297941 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.297861    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"]
Sep 17 19:27:30.298385 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.298206    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kserve-container" containerID="cri-o://650c6fee146cb65eca0c412e83ced4ae82a2f1fa669824fd465da879017be185" gracePeriod=30
Sep 17 19:27:30.298385 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.298238    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kube-rbac-proxy" containerID="cri-o://e274e022aac0f2820ce63b94c8d2ae0fa69764a097d48c8b710386f2df9c53a6" gracePeriod=30
Sep 17 19:27:30.323532 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.323069    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"]
Sep 17 19:27:30.324413 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.324359    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kserve-container" containerID="cri-o://254d13f1353c4d068f50610f665a8e925860380b4fd977f316ea0b6902514186" gracePeriod=30
Sep 17 19:27:30.324852 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.324822    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kube-rbac-proxy" containerID="cri-o://5c63e32e84f6d2b44c430226791af7074457f9b4f8da510ce1b35529ea8866e1" gracePeriod=30
Sep 17 19:27:30.419230 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.419197    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"]
Sep 17 19:27:30.419726 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.419698    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kube-rbac-proxy"
Sep 17 19:27:30.419726 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.419726    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kube-rbac-proxy"
Sep 17 19:27:30.419726 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.419740    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kube-rbac-proxy"
Sep 17 19:27:30.419965 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.419748    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kube-rbac-proxy"
Sep 17 19:27:30.419965 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.419760    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kserve-container"
Sep 17 19:27:30.419965 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.419766    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kserve-container"
Sep 17 19:27:30.419965 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.419785    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kserve-container"
Sep 17 19:27:30.419965 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.419790    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kserve-container"
Sep 17 19:27:30.419965 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.419875    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kserve-container"
Sep 17 19:27:30.419965 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.419888    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bc44fcf-6903-4ce1-988d-2d66fe1da04d" containerName="kube-rbac-proxy"
Sep 17 19:27:30.419965 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.419898    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kube-rbac-proxy"
Sep 17 19:27:30.419965 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.419911    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="600dbcf2-78cf-4eb9-b8b5-95ff97b91539" containerName="kserve-container"
Sep 17 19:27:30.423108 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.423088    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:27:30.426821 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.426801    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-31ec7-kube-rbac-proxy-sar-config\""
Sep 17 19:27:30.430507 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.430482    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-31ec7-predictor-serving-cert\""
Sep 17 19:27:30.435395 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.435374    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"]
Sep 17 19:27:30.536006 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.535974    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"]
Sep 17 19:27:30.539555 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.539538    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:27:30.542936 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.542915    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-31ec7-predictor-serving-cert\""
Sep 17 19:27:30.543218 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.543199    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-31ec7-kube-rbac-proxy-sar-config\""
Sep 17 19:27:30.549402 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.549348    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"]
Sep 17 19:27:30.583133 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.583094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/005c2972-33cc-4d93-a617-b4542865e869-proxy-tls\") pod \"success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6\" (UID: \"005c2972-33cc-4d93-a617-b4542865e869\") " pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:27:30.583317 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.583166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/005c2972-33cc-4d93-a617-b4542865e869-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6\" (UID: \"005c2972-33cc-4d93-a617-b4542865e869\") " pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:27:30.583317 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.583235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9ddg\" (UniqueName: \"kubernetes.io/projected/005c2972-33cc-4d93-a617-b4542865e869-kube-api-access-d9ddg\") pod \"success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6\" (UID: \"005c2972-33cc-4d93-a617-b4542865e869\") " pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:27:30.684639 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.684606    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35869f6f-277a-422e-9cb7-d63e380e437e-proxy-tls\") pod \"error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j\" (UID: \"35869f6f-277a-422e-9cb7-d63e380e437e\") " pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:27:30.684639 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.684654    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/005c2972-33cc-4d93-a617-b4542865e869-proxy-tls\") pod \"success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6\" (UID: \"005c2972-33cc-4d93-a617-b4542865e869\") " pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:27:30.684906 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.684673    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35869f6f-277a-422e-9cb7-d63e380e437e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j\" (UID: \"35869f6f-277a-422e-9cb7-d63e380e437e\") " pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:27:30.684906 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.684714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/005c2972-33cc-4d93-a617-b4542865e869-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6\" (UID: \"005c2972-33cc-4d93-a617-b4542865e869\") " pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:27:30.684906 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.684739    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mrqs\" (UniqueName: \"kubernetes.io/projected/35869f6f-277a-422e-9cb7-d63e380e437e-kube-api-access-2mrqs\") pod \"error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j\" (UID: \"35869f6f-277a-422e-9cb7-d63e380e437e\") " pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:27:30.684906 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.684775    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9ddg\" (UniqueName: \"kubernetes.io/projected/005c2972-33cc-4d93-a617-b4542865e869-kube-api-access-d9ddg\") pod \"success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6\" (UID: \"005c2972-33cc-4d93-a617-b4542865e869\") " pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:27:30.685317 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.685293    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/005c2972-33cc-4d93-a617-b4542865e869-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6\" (UID: \"005c2972-33cc-4d93-a617-b4542865e869\") " pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:27:30.687221 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.687195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/005c2972-33cc-4d93-a617-b4542865e869-proxy-tls\") pod \"success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6\" (UID: \"005c2972-33cc-4d93-a617-b4542865e869\") " pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:27:30.695632 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.695601    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9ddg\" (UniqueName: \"kubernetes.io/projected/005c2972-33cc-4d93-a617-b4542865e869-kube-api-access-d9ddg\") pod \"success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6\" (UID: \"005c2972-33cc-4d93-a617-b4542865e869\") " pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:27:30.733519 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.733483    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:27:30.785264 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.785236    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35869f6f-277a-422e-9cb7-d63e380e437e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j\" (UID: \"35869f6f-277a-422e-9cb7-d63e380e437e\") " pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:27:30.785407 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.785292    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mrqs\" (UniqueName: \"kubernetes.io/projected/35869f6f-277a-422e-9cb7-d63e380e437e-kube-api-access-2mrqs\") pod \"error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j\" (UID: \"35869f6f-277a-422e-9cb7-d63e380e437e\") " pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:27:30.785407 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.785346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35869f6f-277a-422e-9cb7-d63e380e437e-proxy-tls\") pod \"error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j\" (UID: \"35869f6f-277a-422e-9cb7-d63e380e437e\") " pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:27:30.786560 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.786111    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35869f6f-277a-422e-9cb7-d63e380e437e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j\" (UID: \"35869f6f-277a-422e-9cb7-d63e380e437e\") " pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:27:30.788542 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.788475    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35869f6f-277a-422e-9cb7-d63e380e437e-proxy-tls\") pod \"error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j\" (UID: \"35869f6f-277a-422e-9cb7-d63e380e437e\") " pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:27:30.796731 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.796671    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mrqs\" (UniqueName: \"kubernetes.io/projected/35869f6f-277a-422e-9cb7-d63e380e437e-kube-api-access-2mrqs\") pod \"error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j\" (UID: \"35869f6f-277a-422e-9cb7-d63e380e437e\") " pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:27:30.849896 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:30.849807    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:27:31.006263 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.006231    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"]
Sep 17 19:27:31.008119 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:27:31.008097    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005c2972_33cc_4d93_a617_b4542865e869.slice/crio-c237fc90eeab9db9ceba8a9ae715f8b4f05a295be1fb5a69cdb09bb873d23641 WatchSource:0}: Error finding container c237fc90eeab9db9ceba8a9ae715f8b4f05a295be1fb5a69cdb09bb873d23641: Status 404 returned error can't find the container with id c237fc90eeab9db9ceba8a9ae715f8b4f05a295be1fb5a69cdb09bb873d23641
Sep 17 19:27:31.123026 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.122934    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"]
Sep 17 19:27:31.127137 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:27:31.127109    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35869f6f_277a_422e_9cb7_d63e380e437e.slice/crio-1d74f621aba381307ba906bcac29facba217cedc4e21b44aed2e61d62f982a20 WatchSource:0}: Error finding container 1d74f621aba381307ba906bcac29facba217cedc4e21b44aed2e61d62f982a20: Status 404 returned error can't find the container with id 1d74f621aba381307ba906bcac29facba217cedc4e21b44aed2e61d62f982a20
Sep 17 19:27:31.330054 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.330005    2583 generic.go:356] "Generic (PLEG): container finished" podID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerID="e274e022aac0f2820ce63b94c8d2ae0fa69764a097d48c8b710386f2df9c53a6" exitCode=2
Sep 17 19:27:31.330431 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.330123    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" event={"ID":"590eb23d-6765-47e8-8bcd-9e96e06e3e55","Type":"ContainerDied","Data":"e274e022aac0f2820ce63b94c8d2ae0fa69764a097d48c8b710386f2df9c53a6"}
Sep 17 19:27:31.332343 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.332318    2583 generic.go:356] "Generic (PLEG): container finished" podID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerID="5c63e32e84f6d2b44c430226791af7074457f9b4f8da510ce1b35529ea8866e1" exitCode=2
Sep 17 19:27:31.332501 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.332398    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" event={"ID":"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3","Type":"ContainerDied","Data":"5c63e32e84f6d2b44c430226791af7074457f9b4f8da510ce1b35529ea8866e1"}
Sep 17 19:27:31.334487 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.334464    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" event={"ID":"35869f6f-277a-422e-9cb7-d63e380e437e","Type":"ContainerStarted","Data":"4f5673279d2e079a81afe87c195a9a0be7ec51ebec21c2106b6e931bcb868a0a"}
Sep 17 19:27:31.334587 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.334513    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" event={"ID":"35869f6f-277a-422e-9cb7-d63e380e437e","Type":"ContainerStarted","Data":"01160fa5cac79e4d89b438def0e2d02a9df897548b8ca714ce07963881ffb6b3"}
Sep 17 19:27:31.334587 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.334544    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" event={"ID":"35869f6f-277a-422e-9cb7-d63e380e437e","Type":"ContainerStarted","Data":"1d74f621aba381307ba906bcac29facba217cedc4e21b44aed2e61d62f982a20"}
Sep 17 19:27:31.334700 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.334660    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:27:31.334755 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.334702    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:27:31.336153 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.336123    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:27:31.336246 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.336198    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" event={"ID":"005c2972-33cc-4d93-a617-b4542865e869","Type":"ContainerStarted","Data":"45c83dada7f678d968bbe5864cae21237d334b90da29713b0b718289148684e8"}
Sep 17 19:27:31.336246 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.336237    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" event={"ID":"005c2972-33cc-4d93-a617-b4542865e869","Type":"ContainerStarted","Data":"41bd1e0b64f26231b4d15f98079a4648a1f4c9aa4a36111f712dfef9bdd11cd2"}
Sep 17 19:27:31.336320 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.336260    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" event={"ID":"005c2972-33cc-4d93-a617-b4542865e869","Type":"ContainerStarted","Data":"c237fc90eeab9db9ceba8a9ae715f8b4f05a295be1fb5a69cdb09bb873d23641"}
Sep 17 19:27:31.336483 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.336466    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:27:31.336524 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.336514    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:27:31.337529 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.337509    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 19:27:31.354129 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.354079    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" podStartSLOduration=1.354061482 podStartE2EDuration="1.354061482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:27:31.351799948 +0000 UTC m=+1255.179658181" watchObservedRunningTime="2026-09-17 19:27:31.354061482 +0000 UTC m=+1255.181919715"
Sep 17 19:27:31.369165 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:31.369050    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" podStartSLOduration=1.369033414 podStartE2EDuration="1.369033414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:27:31.36752584 +0000 UTC m=+1255.195384074" watchObservedRunningTime="2026-09-17 19:27:31.369033414 +0000 UTC m=+1255.196891647"
Sep 17 19:27:32.340838 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:32.340791    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 19:27:32.341227 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:32.340863    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:27:32.939631 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:32.939583    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 19:27:32.939631 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:32.939617    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 19:27:35.054103 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.054075    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:27:35.225379 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.225340    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrr5g\" (UniqueName: \"kubernetes.io/projected/590eb23d-6765-47e8-8bcd-9e96e06e3e55-kube-api-access-rrr5g\") pod \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\" (UID: \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\") "
Sep 17 19:27:35.225541 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.225421    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/590eb23d-6765-47e8-8bcd-9e96e06e3e55-kube-rbac-proxy-sar-config\") pod \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\" (UID: \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\") "
Sep 17 19:27:35.225541 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.225498    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/590eb23d-6765-47e8-8bcd-9e96e06e3e55-proxy-tls\") pod \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\" (UID: \"590eb23d-6765-47e8-8bcd-9e96e06e3e55\") "
Sep 17 19:27:35.225755 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.225728    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590eb23d-6765-47e8-8bcd-9e96e06e3e55-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "590eb23d-6765-47e8-8bcd-9e96e06e3e55" (UID: "590eb23d-6765-47e8-8bcd-9e96e06e3e55"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:27:35.227518 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.227489    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/590eb23d-6765-47e8-8bcd-9e96e06e3e55-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "590eb23d-6765-47e8-8bcd-9e96e06e3e55" (UID: "590eb23d-6765-47e8-8bcd-9e96e06e3e55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:35.227518 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.227487    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590eb23d-6765-47e8-8bcd-9e96e06e3e55-kube-api-access-rrr5g" (OuterVolumeSpecName: "kube-api-access-rrr5g") pod "590eb23d-6765-47e8-8bcd-9e96e06e3e55" (UID: "590eb23d-6765-47e8-8bcd-9e96e06e3e55"). InnerVolumeSpecName "kube-api-access-rrr5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:35.326556 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.326507    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrr5g\" (UniqueName: \"kubernetes.io/projected/590eb23d-6765-47e8-8bcd-9e96e06e3e55-kube-api-access-rrr5g\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:27:35.326556 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.326549    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/590eb23d-6765-47e8-8bcd-9e96e06e3e55-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:27:35.326556 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.326560    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/590eb23d-6765-47e8-8bcd-9e96e06e3e55-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:27:35.362328 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.362296    2583 generic.go:356] "Generic (PLEG): container finished" podID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerID="650c6fee146cb65eca0c412e83ced4ae82a2f1fa669824fd465da879017be185" exitCode=0
Sep 17 19:27:35.362535 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.362353    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"
Sep 17 19:27:35.362535 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.362380    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" event={"ID":"590eb23d-6765-47e8-8bcd-9e96e06e3e55","Type":"ContainerDied","Data":"650c6fee146cb65eca0c412e83ced4ae82a2f1fa669824fd465da879017be185"}
Sep 17 19:27:35.362535 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.362460    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49" event={"ID":"590eb23d-6765-47e8-8bcd-9e96e06e3e55","Type":"ContainerDied","Data":"c48c8bbd35c45dd384b04ff63ddbb7608c185850a7899efa4592865f131e2c79"}
Sep 17 19:27:35.362535 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.362498    2583 scope.go:117] "RemoveContainer" containerID="e274e022aac0f2820ce63b94c8d2ae0fa69764a097d48c8b710386f2df9c53a6"
Sep 17 19:27:35.364477 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.364452    2583 generic.go:356] "Generic (PLEG): container finished" podID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerID="254d13f1353c4d068f50610f665a8e925860380b4fd977f316ea0b6902514186" exitCode=0
Sep 17 19:27:35.364606 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.364526    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" event={"ID":"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3","Type":"ContainerDied","Data":"254d13f1353c4d068f50610f665a8e925860380b4fd977f316ea0b6902514186"}
Sep 17 19:27:35.372091 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.372070    2583 scope.go:117] "RemoveContainer" containerID="650c6fee146cb65eca0c412e83ced4ae82a2f1fa669824fd465da879017be185"
Sep 17 19:27:35.379983 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.379961    2583 scope.go:117] "RemoveContainer" containerID="e274e022aac0f2820ce63b94c8d2ae0fa69764a097d48c8b710386f2df9c53a6"
Sep 17 19:27:35.380281 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:27:35.380259    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e274e022aac0f2820ce63b94c8d2ae0fa69764a097d48c8b710386f2df9c53a6\": container with ID starting with e274e022aac0f2820ce63b94c8d2ae0fa69764a097d48c8b710386f2df9c53a6 not found: ID does not exist" containerID="e274e022aac0f2820ce63b94c8d2ae0fa69764a097d48c8b710386f2df9c53a6"
Sep 17 19:27:35.380354 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.380295    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e274e022aac0f2820ce63b94c8d2ae0fa69764a097d48c8b710386f2df9c53a6"} err="failed to get container status \"e274e022aac0f2820ce63b94c8d2ae0fa69764a097d48c8b710386f2df9c53a6\": rpc error: code = NotFound desc = could not find container \"e274e022aac0f2820ce63b94c8d2ae0fa69764a097d48c8b710386f2df9c53a6\": container with ID starting with e274e022aac0f2820ce63b94c8d2ae0fa69764a097d48c8b710386f2df9c53a6 not found: ID does not exist"
Sep 17 19:27:35.380354 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.380320    2583 scope.go:117] "RemoveContainer" containerID="650c6fee146cb65eca0c412e83ced4ae82a2f1fa669824fd465da879017be185"
Sep 17 19:27:35.380626 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:27:35.380608    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650c6fee146cb65eca0c412e83ced4ae82a2f1fa669824fd465da879017be185\": container with ID starting with 650c6fee146cb65eca0c412e83ced4ae82a2f1fa669824fd465da879017be185 not found: ID does not exist" containerID="650c6fee146cb65eca0c412e83ced4ae82a2f1fa669824fd465da879017be185"
Sep 17 19:27:35.380666 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.380632    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650c6fee146cb65eca0c412e83ced4ae82a2f1fa669824fd465da879017be185"} err="failed to get container status \"650c6fee146cb65eca0c412e83ced4ae82a2f1fa669824fd465da879017be185\": rpc error: code = NotFound desc = could not find container \"650c6fee146cb65eca0c412e83ced4ae82a2f1fa669824fd465da879017be185\": container with ID starting with 650c6fee146cb65eca0c412e83ced4ae82a2f1fa669824fd465da879017be185 not found: ID does not exist"
Sep 17 19:27:35.385937 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.385914    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"]
Sep 17 19:27:35.389882 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.389857    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-87c35-predictor-77c4c6b6d7-pph49"]
Sep 17 19:27:35.540204 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.540181    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:27:35.628698 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.628614    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf549\" (UniqueName: \"kubernetes.io/projected/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-kube-api-access-pf549\") pod \"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3\" (UID: \"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3\") "
Sep 17 19:27:35.628698 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.628651    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-kube-rbac-proxy-sar-config\") pod \"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3\" (UID: \"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3\") "
Sep 17 19:27:35.628698 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.628693    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-proxy-tls\") pod \"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3\" (UID: \"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3\") "
Sep 17 19:27:35.629148 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.629118    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" (UID: "c801e156-bc4a-4c8f-8a91-bce79f1ca6b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:27:35.630813 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.630783    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" (UID: "c801e156-bc4a-4c8f-8a91-bce79f1ca6b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:35.630813 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.630795    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-kube-api-access-pf549" (OuterVolumeSpecName: "kube-api-access-pf549") pod "c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" (UID: "c801e156-bc4a-4c8f-8a91-bce79f1ca6b3"). InnerVolumeSpecName "kube-api-access-pf549". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:35.730095 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.730058    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:27:35.730095 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.730089    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf549\" (UniqueName: \"kubernetes.io/projected/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-kube-api-access-pf549\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:27:35.730095 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:35.730100    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:27:36.370525 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:36.370499    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"
Sep 17 19:27:36.370969 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:36.370532    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn" event={"ID":"c801e156-bc4a-4c8f-8a91-bce79f1ca6b3","Type":"ContainerDied","Data":"fc9a1af7ac3068626b50b2af6bca212e94a559877c3ee8eafafd8d8e7cdbfa4c"}
Sep 17 19:27:36.370969 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:36.370610    2583 scope.go:117] "RemoveContainer" containerID="5c63e32e84f6d2b44c430226791af7074457f9b4f8da510ce1b35529ea8866e1"
Sep 17 19:27:36.379369 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:36.379348    2583 scope.go:117] "RemoveContainer" containerID="254d13f1353c4d068f50610f665a8e925860380b4fd977f316ea0b6902514186"
Sep 17 19:27:36.391977 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:36.391938    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"]
Sep 17 19:27:36.395266 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:36.395242    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-87c35-predictor-586c54bbcd-cdtvn"]
Sep 17 19:27:36.452927 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:36.452892    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" path="/var/lib/kubelet/pods/590eb23d-6765-47e8-8bcd-9e96e06e3e55/volumes"
Sep 17 19:27:36.453410 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:36.453389    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" path="/var/lib/kubelet/pods/c801e156-bc4a-4c8f-8a91-bce79f1ca6b3/volumes"
Sep 17 19:27:37.345514 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:37.345484    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:27:37.345717 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:37.345575    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:27:37.345950 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:37.345924    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:27:37.346269 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:37.346246    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 19:27:38.248165 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:38.248128    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 19:27:38.251421 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:38.251392    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 19:27:47.346308 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:47.346268    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:27:47.346783 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:47.346270    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 19:27:48.248399 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:48.248358    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 19:27:48.251679 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:48.251653    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 19:27:57.346271 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:57.346214    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:27:57.346681 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:57.346216    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 19:27:58.248923 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:58.248883    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 19:27:58.252180 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:27:58.252155    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 19:28:07.346022 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:07.345975    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:28:07.346426 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:07.346198    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 19:28:08.249611 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:08.249580    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:28:08.251604 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:08.251578    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:28:17.345964 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:17.345925    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:28:17.346459 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:17.346202    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 19:28:27.347606 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:27.347575    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:28:27.348261 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:27.347671    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:28:31.687633 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.687602    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"]
Sep 17 19:28:31.688041 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.687863    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kserve-container" containerID="cri-o://85bd81ef19c81d63294889c8f1633e559524878974d59ec66381f829e06872af" gracePeriod=30
Sep 17 19:28:31.688041 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.687998    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kube-rbac-proxy" containerID="cri-o://4d4d27606b15460dc92908384998319243ccfb65adc7e156f2d332ff44af977b" gracePeriod=30
Sep 17 19:28:31.727661 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.727625    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"]
Sep 17 19:28:31.727956 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.727929    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kserve-container" containerID="cri-o://21cede2f8c7a03ae55dc9b017dda43328647f67d4a3ec88c1b1eff0b6ebdc056" gracePeriod=30
Sep 17 19:28:31.728022 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.727945    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kube-rbac-proxy" containerID="cri-o://71018ad550c583828721c83d47af13220eeaf565c7758e5fbd77994ec382f6e7" gracePeriod=30
Sep 17 19:28:31.813378 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.813345    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"]
Sep 17 19:28:31.813772 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.813757    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kube-rbac-proxy"
Sep 17 19:28:31.813832 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.813772    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kube-rbac-proxy"
Sep 17 19:28:31.813832 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.813782    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kserve-container"
Sep 17 19:28:31.813832 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.813787    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kserve-container"
Sep 17 19:28:31.813832 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.813801    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kube-rbac-proxy"
Sep 17 19:28:31.813832 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.813806    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kube-rbac-proxy"
Sep 17 19:28:31.813832 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.813815    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kserve-container"
Sep 17 19:28:31.813832 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.813821    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kserve-container"
Sep 17 19:28:31.814038 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.813876    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kube-rbac-proxy"
Sep 17 19:28:31.814038 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.813883    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kube-rbac-proxy"
Sep 17 19:28:31.814038 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.813894    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="590eb23d-6765-47e8-8bcd-9e96e06e3e55" containerName="kserve-container"
Sep 17 19:28:31.814038 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.813900    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c801e156-bc4a-4c8f-8a91-bce79f1ca6b3" containerName="kserve-container"
Sep 17 19:28:31.816851 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.816835    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:28:31.819414 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.819389    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cedb0-kube-rbac-proxy-sar-config\""
Sep 17 19:28:31.819562 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.819389    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cedb0-predictor-serving-cert\""
Sep 17 19:28:31.827057 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.827031    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"]
Sep 17 19:28:31.918700 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.918662    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"]
Sep 17 19:28:31.926240 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.926207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4168fdd-5d8b-4a40-a773-5ab359e5e344-proxy-tls\") pod \"success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw\" (UID: \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\") " pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:28:31.926382 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.926285    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4168fdd-5d8b-4a40-a773-5ab359e5e344-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw\" (UID: \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\") " pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:28:31.926382 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.926322    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxnnj\" (UniqueName: \"kubernetes.io/projected/f4168fdd-5d8b-4a40-a773-5ab359e5e344-kube-api-access-cxnnj\") pod \"success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw\" (UID: \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\") " pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:28:31.931650 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.931624    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"]
Sep 17 19:28:31.931777 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.931729    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:28:31.934668 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.934643    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cedb0-predictor-serving-cert\""
Sep 17 19:28:31.934810 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:31.934670    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cedb0-kube-rbac-proxy-sar-config\""
Sep 17 19:28:32.027267 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.027234    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4168fdd-5d8b-4a40-a773-5ab359e5e344-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw\" (UID: \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\") " pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:28:32.027267 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.027268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxnnj\" (UniqueName: \"kubernetes.io/projected/f4168fdd-5d8b-4a40-a773-5ab359e5e344-kube-api-access-cxnnj\") pod \"success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw\" (UID: \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\") " pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:28:32.027587 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.027414    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38870954-f193-4fe8-84ea-5a6ff226c37e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cedb0-predictor-57c4b47646-c9hsm\" (UID: \"38870954-f193-4fe8-84ea-5a6ff226c37e\") " pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:28:32.027587 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.027537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4168fdd-5d8b-4a40-a773-5ab359e5e344-proxy-tls\") pod \"success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw\" (UID: \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\") " pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:28:32.027689 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:28:32.027602    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-serving-cert: secret "success-200-isvc-cedb0-predictor-serving-cert" not found
Sep 17 19:28:32.027689 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.027608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38870954-f193-4fe8-84ea-5a6ff226c37e-proxy-tls\") pod \"error-404-isvc-cedb0-predictor-57c4b47646-c9hsm\" (UID: \"38870954-f193-4fe8-84ea-5a6ff226c37e\") " pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:28:32.027689 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.027642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljd9s\" (UniqueName: \"kubernetes.io/projected/38870954-f193-4fe8-84ea-5a6ff226c37e-kube-api-access-ljd9s\") pod \"error-404-isvc-cedb0-predictor-57c4b47646-c9hsm\" (UID: \"38870954-f193-4fe8-84ea-5a6ff226c37e\") " pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:28:32.027689 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:28:32.027673    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4168fdd-5d8b-4a40-a773-5ab359e5e344-proxy-tls podName:f4168fdd-5d8b-4a40-a773-5ab359e5e344 nodeName:}" failed. No retries permitted until 2026-09-17 19:28:32.527659627 +0000 UTC m=+1316.355517844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f4168fdd-5d8b-4a40-a773-5ab359e5e344-proxy-tls") pod "success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" (UID: "f4168fdd-5d8b-4a40-a773-5ab359e5e344") : secret "success-200-isvc-cedb0-predictor-serving-cert" not found
Sep 17 19:28:32.027989 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.027971    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4168fdd-5d8b-4a40-a773-5ab359e5e344-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw\" (UID: \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\") " pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:28:32.038026 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.038003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxnnj\" (UniqueName: \"kubernetes.io/projected/f4168fdd-5d8b-4a40-a773-5ab359e5e344-kube-api-access-cxnnj\") pod \"success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw\" (UID: \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\") " pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:28:32.128497 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.128435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38870954-f193-4fe8-84ea-5a6ff226c37e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cedb0-predictor-57c4b47646-c9hsm\" (UID: \"38870954-f193-4fe8-84ea-5a6ff226c37e\") " pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:28:32.128760 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.128551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38870954-f193-4fe8-84ea-5a6ff226c37e-proxy-tls\") pod \"error-404-isvc-cedb0-predictor-57c4b47646-c9hsm\" (UID: \"38870954-f193-4fe8-84ea-5a6ff226c37e\") " pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:28:32.128760 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.128579    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljd9s\" (UniqueName: \"kubernetes.io/projected/38870954-f193-4fe8-84ea-5a6ff226c37e-kube-api-access-ljd9s\") pod \"error-404-isvc-cedb0-predictor-57c4b47646-c9hsm\" (UID: \"38870954-f193-4fe8-84ea-5a6ff226c37e\") " pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:28:32.129094 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.129072    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38870954-f193-4fe8-84ea-5a6ff226c37e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cedb0-predictor-57c4b47646-c9hsm\" (UID: \"38870954-f193-4fe8-84ea-5a6ff226c37e\") " pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:28:32.131039 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.131015    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38870954-f193-4fe8-84ea-5a6ff226c37e-proxy-tls\") pod \"error-404-isvc-cedb0-predictor-57c4b47646-c9hsm\" (UID: \"38870954-f193-4fe8-84ea-5a6ff226c37e\") " pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:28:32.137162 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.137130    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljd9s\" (UniqueName: \"kubernetes.io/projected/38870954-f193-4fe8-84ea-5a6ff226c37e-kube-api-access-ljd9s\") pod \"error-404-isvc-cedb0-predictor-57c4b47646-c9hsm\" (UID: \"38870954-f193-4fe8-84ea-5a6ff226c37e\") " pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:28:32.242686 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.242597    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:28:32.514167 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.514086    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"]
Sep 17 19:28:32.518211 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:28:32.518012    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38870954_f193_4fe8_84ea_5a6ff226c37e.slice/crio-e9c20199b378ca2366a18e86c4f3acedb96846edcf8e30c06088eeebdf0420d4 WatchSource:0}: Error finding container e9c20199b378ca2366a18e86c4f3acedb96846edcf8e30c06088eeebdf0420d4: Status 404 returned error can't find the container with id e9c20199b378ca2366a18e86c4f3acedb96846edcf8e30c06088eeebdf0420d4
Sep 17 19:28:32.532065 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.532043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4168fdd-5d8b-4a40-a773-5ab359e5e344-proxy-tls\") pod \"success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw\" (UID: \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\") " pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:28:32.534435 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.534413    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4168fdd-5d8b-4a40-a773-5ab359e5e344-proxy-tls\") pod \"success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw\" (UID: \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\") " pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:28:32.632705 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.632679    2583 generic.go:356] "Generic (PLEG): container finished" podID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerID="71018ad550c583828721c83d47af13220eeaf565c7758e5fbd77994ec382f6e7" exitCode=2
Sep 17 19:28:32.632826 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.632754    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" event={"ID":"631d021b-4a9a-4b39-80a6-54cf8b103db4","Type":"ContainerDied","Data":"71018ad550c583828721c83d47af13220eeaf565c7758e5fbd77994ec382f6e7"}
Sep 17 19:28:32.634422 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.634402    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" event={"ID":"38870954-f193-4fe8-84ea-5a6ff226c37e","Type":"ContainerStarted","Data":"64a4f2295a5e6e5458a21d874b4f9cf0c8ae89f1cb0cccd2447886e44a4c9613"}
Sep 17 19:28:32.634613 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.634458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" event={"ID":"38870954-f193-4fe8-84ea-5a6ff226c37e","Type":"ContainerStarted","Data":"e9c20199b378ca2366a18e86c4f3acedb96846edcf8e30c06088eeebdf0420d4"}
Sep 17 19:28:32.634613 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.634497    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:28:32.634729 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.634643    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:28:32.635896 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.635874    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 19:28:32.636376 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.636349    2583 generic.go:356] "Generic (PLEG): container finished" podID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerID="4d4d27606b15460dc92908384998319243ccfb65adc7e156f2d332ff44af977b" exitCode=2
Sep 17 19:28:32.636465 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.636416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" event={"ID":"14e323e0-7081-47e9-9bc8-fae817f75bbe","Type":"ContainerDied","Data":"4d4d27606b15460dc92908384998319243ccfb65adc7e156f2d332ff44af977b"}
Sep 17 19:28:32.654733 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.654679    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" podStartSLOduration=1.654663181 podStartE2EDuration="1.654663181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:28:32.652565576 +0000 UTC m=+1316.480423809" watchObservedRunningTime="2026-09-17 19:28:32.654663181 +0000 UTC m=+1316.482521415"
Sep 17 19:28:32.729477 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:32.729427    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:28:33.002231 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:33.002209    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"]
Sep 17 19:28:33.004266 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:28:33.004227    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4168fdd_5d8b_4a40_a773_5ab359e5e344.slice/crio-3cbc1e07c2b670e49b3fecd3c3e59a5f12fd75c09a9aef0f2949e78e90869130 WatchSource:0}: Error finding container 3cbc1e07c2b670e49b3fecd3c3e59a5f12fd75c09a9aef0f2949e78e90869130: Status 404 returned error can't find the container with id 3cbc1e07c2b670e49b3fecd3c3e59a5f12fd75c09a9aef0f2949e78e90869130
Sep 17 19:28:33.240373 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:33.240328    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 17 19:28:33.240564 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:33.240328    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 17 19:28:33.643690 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:33.643654    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" event={"ID":"f4168fdd-5d8b-4a40-a773-5ab359e5e344","Type":"ContainerStarted","Data":"63c3cce2d2ad037c5b7571e026d3c2f9b607f089e7bdf043042125c1e5f97c5f"}
Sep 17 19:28:33.643927 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:33.643725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" event={"ID":"f4168fdd-5d8b-4a40-a773-5ab359e5e344","Type":"ContainerStarted","Data":"13f084db19c537f41a8ee15bf7874d36b1414d75c65046cec6774d69e963a33b"}
Sep 17 19:28:33.643927 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:33.643849    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:28:33.643927 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:33.643907    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:28:33.644089 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:33.643939    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" event={"ID":"f4168fdd-5d8b-4a40-a773-5ab359e5e344","Type":"ContainerStarted","Data":"3cbc1e07c2b670e49b3fecd3c3e59a5f12fd75c09a9aef0f2949e78e90869130"}
Sep 17 19:28:33.645618 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:33.645590    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:28:33.645755 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:33.645671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" event={"ID":"38870954-f193-4fe8-84ea-5a6ff226c37e","Type":"ContainerStarted","Data":"947aa6c88adfe998ec33219e479144d95cf1ce3c877aa901c2465ac390b3a41a"}
Sep 17 19:28:33.645973 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:33.645953    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 19:28:33.662052 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:33.661995    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" podStartSLOduration=2.661976477 podStartE2EDuration="2.661976477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:28:33.661701046 +0000 UTC m=+1317.489559281" watchObservedRunningTime="2026-09-17 19:28:33.661976477 +0000 UTC m=+1317.489834712"
Sep 17 19:28:34.650433 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:34.650393    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 19:28:34.650923 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:34.650518    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:28:35.658505 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:35.658461    2583 generic.go:356] "Generic (PLEG): container finished" podID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerID="85bd81ef19c81d63294889c8f1633e559524878974d59ec66381f829e06872af" exitCode=0
Sep 17 19:28:35.658505 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:35.658484    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" event={"ID":"14e323e0-7081-47e9-9bc8-fae817f75bbe","Type":"ContainerDied","Data":"85bd81ef19c81d63294889c8f1633e559524878974d59ec66381f829e06872af"}
Sep 17 19:28:35.752141 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:35.752115    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:28:35.862838 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:35.862807    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14e323e0-7081-47e9-9bc8-fae817f75bbe-kube-rbac-proxy-sar-config\") pod \"14e323e0-7081-47e9-9bc8-fae817f75bbe\" (UID: \"14e323e0-7081-47e9-9bc8-fae817f75bbe\") "
Sep 17 19:28:35.863037 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:35.862887    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktkxq\" (UniqueName: \"kubernetes.io/projected/14e323e0-7081-47e9-9bc8-fae817f75bbe-kube-api-access-ktkxq\") pod \"14e323e0-7081-47e9-9bc8-fae817f75bbe\" (UID: \"14e323e0-7081-47e9-9bc8-fae817f75bbe\") "
Sep 17 19:28:35.863037 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:35.862936    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14e323e0-7081-47e9-9bc8-fae817f75bbe-proxy-tls\") pod \"14e323e0-7081-47e9-9bc8-fae817f75bbe\" (UID: \"14e323e0-7081-47e9-9bc8-fae817f75bbe\") "
Sep 17 19:28:35.863180 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:35.863156    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e323e0-7081-47e9-9bc8-fae817f75bbe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "14e323e0-7081-47e9-9bc8-fae817f75bbe" (UID: "14e323e0-7081-47e9-9bc8-fae817f75bbe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:28:35.865023 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:35.864994    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e323e0-7081-47e9-9bc8-fae817f75bbe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "14e323e0-7081-47e9-9bc8-fae817f75bbe" (UID: "14e323e0-7081-47e9-9bc8-fae817f75bbe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:28:35.865023 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:35.864999    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e323e0-7081-47e9-9bc8-fae817f75bbe-kube-api-access-ktkxq" (OuterVolumeSpecName: "kube-api-access-ktkxq") pod "14e323e0-7081-47e9-9bc8-fae817f75bbe" (UID: "14e323e0-7081-47e9-9bc8-fae817f75bbe"). InnerVolumeSpecName "kube-api-access-ktkxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:28:35.963723 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:35.963668    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktkxq\" (UniqueName: \"kubernetes.io/projected/14e323e0-7081-47e9-9bc8-fae817f75bbe-kube-api-access-ktkxq\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:28:35.963723 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:35.963716    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14e323e0-7081-47e9-9bc8-fae817f75bbe-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:28:35.963723 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:35.963726    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14e323e0-7081-47e9-9bc8-fae817f75bbe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:28:36.569587 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.569562    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:28:36.664924 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.664886    2583 generic.go:356] "Generic (PLEG): container finished" podID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerID="21cede2f8c7a03ae55dc9b017dda43328647f67d4a3ec88c1b1eff0b6ebdc056" exitCode=0
Sep 17 19:28:36.665353 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.664936    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"
Sep 17 19:28:36.665353 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.664944    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" event={"ID":"631d021b-4a9a-4b39-80a6-54cf8b103db4","Type":"ContainerDied","Data":"21cede2f8c7a03ae55dc9b017dda43328647f67d4a3ec88c1b1eff0b6ebdc056"}
Sep 17 19:28:36.665353 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.665008    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824" event={"ID":"631d021b-4a9a-4b39-80a6-54cf8b103db4","Type":"ContainerDied","Data":"08fdf15706bb365cf1b2bc2eb1f8c43ac6251bedff375b0910666486ed65ee98"}
Sep 17 19:28:36.665353 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.665036    2583 scope.go:117] "RemoveContainer" containerID="71018ad550c583828721c83d47af13220eeaf565c7758e5fbd77994ec382f6e7"
Sep 17 19:28:36.666760 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.666731    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn" event={"ID":"14e323e0-7081-47e9-9bc8-fae817f75bbe","Type":"ContainerDied","Data":"66028b5733f95409c925f7fc414743e9c6ecf1d7723d29d0d2efde1f9217b795"}
Sep 17 19:28:36.666760 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.666751    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"
Sep 17 19:28:36.670123 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.670102    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcptg\" (UniqueName: \"kubernetes.io/projected/631d021b-4a9a-4b39-80a6-54cf8b103db4-kube-api-access-tcptg\") pod \"631d021b-4a9a-4b39-80a6-54cf8b103db4\" (UID: \"631d021b-4a9a-4b39-80a6-54cf8b103db4\") "
Sep 17 19:28:36.670302 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.670286    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/631d021b-4a9a-4b39-80a6-54cf8b103db4-kube-rbac-proxy-sar-config\") pod \"631d021b-4a9a-4b39-80a6-54cf8b103db4\" (UID: \"631d021b-4a9a-4b39-80a6-54cf8b103db4\") "
Sep 17 19:28:36.670373 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.670358    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/631d021b-4a9a-4b39-80a6-54cf8b103db4-proxy-tls\") pod \"631d021b-4a9a-4b39-80a6-54cf8b103db4\" (UID: \"631d021b-4a9a-4b39-80a6-54cf8b103db4\") "
Sep 17 19:28:36.670661 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.670640    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631d021b-4a9a-4b39-80a6-54cf8b103db4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "631d021b-4a9a-4b39-80a6-54cf8b103db4" (UID: "631d021b-4a9a-4b39-80a6-54cf8b103db4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:28:36.672103 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.672079    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631d021b-4a9a-4b39-80a6-54cf8b103db4-kube-api-access-tcptg" (OuterVolumeSpecName: "kube-api-access-tcptg") pod "631d021b-4a9a-4b39-80a6-54cf8b103db4" (UID: "631d021b-4a9a-4b39-80a6-54cf8b103db4"). InnerVolumeSpecName "kube-api-access-tcptg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:28:36.672203 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.672123    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631d021b-4a9a-4b39-80a6-54cf8b103db4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "631d021b-4a9a-4b39-80a6-54cf8b103db4" (UID: "631d021b-4a9a-4b39-80a6-54cf8b103db4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:28:36.676587 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.676569    2583 scope.go:117] "RemoveContainer" containerID="21cede2f8c7a03ae55dc9b017dda43328647f67d4a3ec88c1b1eff0b6ebdc056"
Sep 17 19:28:36.683232 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.683203    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"]
Sep 17 19:28:36.687775 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.687752    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-30eba-predictor-68c87bd657-pdbdn"]
Sep 17 19:28:36.690031 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.690014    2583 scope.go:117] "RemoveContainer" containerID="71018ad550c583828721c83d47af13220eeaf565c7758e5fbd77994ec382f6e7"
Sep 17 19:28:36.690321 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:28:36.690302    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71018ad550c583828721c83d47af13220eeaf565c7758e5fbd77994ec382f6e7\": container with ID starting with 71018ad550c583828721c83d47af13220eeaf565c7758e5fbd77994ec382f6e7 not found: ID does not exist" containerID="71018ad550c583828721c83d47af13220eeaf565c7758e5fbd77994ec382f6e7"
Sep 17 19:28:36.690395 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.690328    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71018ad550c583828721c83d47af13220eeaf565c7758e5fbd77994ec382f6e7"} err="failed to get container status \"71018ad550c583828721c83d47af13220eeaf565c7758e5fbd77994ec382f6e7\": rpc error: code = NotFound desc = could not find container \"71018ad550c583828721c83d47af13220eeaf565c7758e5fbd77994ec382f6e7\": container with ID starting with 71018ad550c583828721c83d47af13220eeaf565c7758e5fbd77994ec382f6e7 not found: ID does not exist"
Sep 17 19:28:36.690395 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.690351    2583 scope.go:117] "RemoveContainer" containerID="21cede2f8c7a03ae55dc9b017dda43328647f67d4a3ec88c1b1eff0b6ebdc056"
Sep 17 19:28:36.690661 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:28:36.690623    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21cede2f8c7a03ae55dc9b017dda43328647f67d4a3ec88c1b1eff0b6ebdc056\": container with ID starting with 21cede2f8c7a03ae55dc9b017dda43328647f67d4a3ec88c1b1eff0b6ebdc056 not found: ID does not exist" containerID="21cede2f8c7a03ae55dc9b017dda43328647f67d4a3ec88c1b1eff0b6ebdc056"
Sep 17 19:28:36.690661 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.690649    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21cede2f8c7a03ae55dc9b017dda43328647f67d4a3ec88c1b1eff0b6ebdc056"} err="failed to get container status \"21cede2f8c7a03ae55dc9b017dda43328647f67d4a3ec88c1b1eff0b6ebdc056\": rpc error: code = NotFound desc = could not find container \"21cede2f8c7a03ae55dc9b017dda43328647f67d4a3ec88c1b1eff0b6ebdc056\": container with ID starting with 21cede2f8c7a03ae55dc9b017dda43328647f67d4a3ec88c1b1eff0b6ebdc056 not found: ID does not exist"
Sep 17 19:28:36.690770 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.690663    2583 scope.go:117] "RemoveContainer" containerID="4d4d27606b15460dc92908384998319243ccfb65adc7e156f2d332ff44af977b"
Sep 17 19:28:36.698312 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.698296    2583 scope.go:117] "RemoveContainer" containerID="85bd81ef19c81d63294889c8f1633e559524878974d59ec66381f829e06872af"
Sep 17 19:28:36.771865 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.771830    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcptg\" (UniqueName: \"kubernetes.io/projected/631d021b-4a9a-4b39-80a6-54cf8b103db4-kube-api-access-tcptg\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:28:36.771865 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.771857    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/631d021b-4a9a-4b39-80a6-54cf8b103db4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:28:36.771865 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.771867    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/631d021b-4a9a-4b39-80a6-54cf8b103db4-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:28:36.987169 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.987137    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"]
Sep 17 19:28:36.992251 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:36.992229    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-30eba-predictor-8cb6986c6-fk824"]
Sep 17 19:28:38.452989 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:38.452961    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" path="/var/lib/kubelet/pods/14e323e0-7081-47e9-9bc8-fae817f75bbe/volumes"
Sep 17 19:28:38.453377 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:38.453363    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" path="/var/lib/kubelet/pods/631d021b-4a9a-4b39-80a6-54cf8b103db4/volumes"
Sep 17 19:28:39.655235 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:39.655203    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:28:39.655712 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:39.655565    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:28:39.655712 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:39.655682    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 19:28:39.655973 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:39.655946    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:28:49.656231 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:49.656147    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:28:49.656615 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:49.656146    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 19:28:59.656186 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:59.656144    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:28:59.656753 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:28:59.656145    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 19:29:09.656273 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:29:09.656236    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:29:09.656678 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:29:09.656232    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 19:29:19.656331 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:29:19.656291    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:29:19.656331 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:29:19.656291    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 19:29:29.656638 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:29:29.656606    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:29:29.659134 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:29:29.656982    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:31:38.448065 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:31:38.448033    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:31:38.452490 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:31:38.452468    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:36:38.479103 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:38.479074    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:36:38.485157 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:38.485138    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:36:55.877015 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:55.876988    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"]
Sep 17 19:36:55.877539 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:55.877203    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kserve-container" containerID="cri-o://41bd1e0b64f26231b4d15f98079a4648a1f4c9aa4a36111f712dfef9bdd11cd2" gracePeriod=30
Sep 17 19:36:55.877539 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:55.877226    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kube-rbac-proxy" containerID="cri-o://45c83dada7f678d968bbe5864cae21237d334b90da29713b0b718289148684e8" gracePeriod=30
Sep 17 19:36:55.916886 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:55.916859    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"]
Sep 17 19:36:55.917094 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:55.917068    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kserve-container" containerID="cri-o://01160fa5cac79e4d89b438def0e2d02a9df897548b8ca714ce07963881ffb6b3" gracePeriod=30
Sep 17 19:36:55.917172 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:55.917090    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kube-rbac-proxy" containerID="cri-o://4f5673279d2e079a81afe87c195a9a0be7ec51ebec21c2106b6e931bcb868a0a" gracePeriod=30
Sep 17 19:36:56.061364 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.061340    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"]
Sep 17 19:36:56.061719 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.061707    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kserve-container"
Sep 17 19:36:56.061772 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.061720    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kserve-container"
Sep 17 19:36:56.061772 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.061736    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kube-rbac-proxy"
Sep 17 19:36:56.061772 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.061742    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kube-rbac-proxy"
Sep 17 19:36:56.061772 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.061754    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kube-rbac-proxy"
Sep 17 19:36:56.061772 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.061760    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kube-rbac-proxy"
Sep 17 19:36:56.061772 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.061768    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kserve-container"
Sep 17 19:36:56.061772 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.061773    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kserve-container"
Sep 17 19:36:56.062000 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.061839    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kserve-container"
Sep 17 19:36:56.062000 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.061847    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kube-rbac-proxy"
Sep 17 19:36:56.062000 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.061855    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="14e323e0-7081-47e9-9bc8-fae817f75bbe" containerName="kube-rbac-proxy"
Sep 17 19:36:56.062000 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.061860    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="631d021b-4a9a-4b39-80a6-54cf8b103db4" containerName="kserve-container"
Sep 17 19:36:56.064758 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.064744    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:36:56.067566 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.067548    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fe2a8-predictor-serving-cert\""
Sep 17 19:36:56.067692 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.067675    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fe2a8-kube-rbac-proxy-sar-config\""
Sep 17 19:36:56.078188 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.078122    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"]
Sep 17 19:36:56.106415 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.106392    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b188c85-659e-4241-8c59-f2d68a54da68-proxy-tls\") pod \"success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w\" (UID: \"0b188c85-659e-4241-8c59-f2d68a54da68\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:36:56.106549 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.106476    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b188c85-659e-4241-8c59-f2d68a54da68-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w\" (UID: \"0b188c85-659e-4241-8c59-f2d68a54da68\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:36:56.106549 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.106514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smk9b\" (UniqueName: \"kubernetes.io/projected/0b188c85-659e-4241-8c59-f2d68a54da68-kube-api-access-smk9b\") pod \"success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w\" (UID: \"0b188c85-659e-4241-8c59-f2d68a54da68\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:36:56.129276 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.129211    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"]
Sep 17 19:36:56.132788 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.132773    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:36:56.135478 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.135431    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fe2a8-predictor-serving-cert\""
Sep 17 19:36:56.135580 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.135499    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fe2a8-kube-rbac-proxy-sar-config\""
Sep 17 19:36:56.141550 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.141531    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"]
Sep 17 19:36:56.207754 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.207723    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b188c85-659e-4241-8c59-f2d68a54da68-proxy-tls\") pod \"success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w\" (UID: \"0b188c85-659e-4241-8c59-f2d68a54da68\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:36:56.207891 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.207763    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536aa7c2-08cf-4a18-b895-c4c8997acafb-proxy-tls\") pod \"error-404-isvc-fe2a8-predictor-68889756d6-6mwlj\" (UID: \"536aa7c2-08cf-4a18-b895-c4c8997acafb\") " pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:36:56.207891 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.207799    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536aa7c2-08cf-4a18-b895-c4c8997acafb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fe2a8-predictor-68889756d6-6mwlj\" (UID: \"536aa7c2-08cf-4a18-b895-c4c8997acafb\") " pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:36:56.207891 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.207830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b188c85-659e-4241-8c59-f2d68a54da68-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w\" (UID: \"0b188c85-659e-4241-8c59-f2d68a54da68\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:36:56.207891 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.207857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smk9b\" (UniqueName: \"kubernetes.io/projected/0b188c85-659e-4241-8c59-f2d68a54da68-kube-api-access-smk9b\") pod \"success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w\" (UID: \"0b188c85-659e-4241-8c59-f2d68a54da68\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:36:56.207891 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:36:56.207858    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-serving-cert: secret "success-200-isvc-fe2a8-predictor-serving-cert" not found
Sep 17 19:36:56.208159 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:36:56.207999    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b188c85-659e-4241-8c59-f2d68a54da68-proxy-tls podName:0b188c85-659e-4241-8c59-f2d68a54da68 nodeName:}" failed. No retries permitted until 2026-09-17 19:36:56.70798179 +0000 UTC m=+1820.535840003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0b188c85-659e-4241-8c59-f2d68a54da68-proxy-tls") pod "success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" (UID: "0b188c85-659e-4241-8c59-f2d68a54da68") : secret "success-200-isvc-fe2a8-predictor-serving-cert" not found
Sep 17 19:36:56.208159 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.208044    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghj45\" (UniqueName: \"kubernetes.io/projected/536aa7c2-08cf-4a18-b895-c4c8997acafb-kube-api-access-ghj45\") pod \"error-404-isvc-fe2a8-predictor-68889756d6-6mwlj\" (UID: \"536aa7c2-08cf-4a18-b895-c4c8997acafb\") " pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:36:56.208589 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.208569    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b188c85-659e-4241-8c59-f2d68a54da68-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w\" (UID: \"0b188c85-659e-4241-8c59-f2d68a54da68\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:36:56.216648 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.216626    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smk9b\" (UniqueName: \"kubernetes.io/projected/0b188c85-659e-4241-8c59-f2d68a54da68-kube-api-access-smk9b\") pod \"success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w\" (UID: \"0b188c85-659e-4241-8c59-f2d68a54da68\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:36:56.309166 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.309132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghj45\" (UniqueName: \"kubernetes.io/projected/536aa7c2-08cf-4a18-b895-c4c8997acafb-kube-api-access-ghj45\") pod \"error-404-isvc-fe2a8-predictor-68889756d6-6mwlj\" (UID: \"536aa7c2-08cf-4a18-b895-c4c8997acafb\") " pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:36:56.309344 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.309192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536aa7c2-08cf-4a18-b895-c4c8997acafb-proxy-tls\") pod \"error-404-isvc-fe2a8-predictor-68889756d6-6mwlj\" (UID: \"536aa7c2-08cf-4a18-b895-c4c8997acafb\") " pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:36:56.309344 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.309230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536aa7c2-08cf-4a18-b895-c4c8997acafb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fe2a8-predictor-68889756d6-6mwlj\" (UID: \"536aa7c2-08cf-4a18-b895-c4c8997acafb\") " pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:36:56.309858 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.309838    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536aa7c2-08cf-4a18-b895-c4c8997acafb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fe2a8-predictor-68889756d6-6mwlj\" (UID: \"536aa7c2-08cf-4a18-b895-c4c8997acafb\") " pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:36:56.311620 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.311600    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536aa7c2-08cf-4a18-b895-c4c8997acafb-proxy-tls\") pod \"error-404-isvc-fe2a8-predictor-68889756d6-6mwlj\" (UID: \"536aa7c2-08cf-4a18-b895-c4c8997acafb\") " pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:36:56.317354 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.317333    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghj45\" (UniqueName: \"kubernetes.io/projected/536aa7c2-08cf-4a18-b895-c4c8997acafb-kube-api-access-ghj45\") pod \"error-404-isvc-fe2a8-predictor-68889756d6-6mwlj\" (UID: \"536aa7c2-08cf-4a18-b895-c4c8997acafb\") " pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:36:56.443487 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.443375    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:36:56.713392 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.713366    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"]
Sep 17 19:36:56.713485 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.713464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b188c85-659e-4241-8c59-f2d68a54da68-proxy-tls\") pod \"success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w\" (UID: \"0b188c85-659e-4241-8c59-f2d68a54da68\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:36:56.715800 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:36:56.715773    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536aa7c2_08cf_4a18_b895_c4c8997acafb.slice/crio-d2e936c3d3125b01c3b4c4d5644c0bd5ea0bdcb1db5767073c67f010c30ea7f1 WatchSource:0}: Error finding container d2e936c3d3125b01c3b4c4d5644c0bd5ea0bdcb1db5767073c67f010c30ea7f1: Status 404 returned error can't find the container with id d2e936c3d3125b01c3b4c4d5644c0bd5ea0bdcb1db5767073c67f010c30ea7f1
Sep 17 19:36:56.715926 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.715876    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b188c85-659e-4241-8c59-f2d68a54da68-proxy-tls\") pod \"success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w\" (UID: \"0b188c85-659e-4241-8c59-f2d68a54da68\") " pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:36:56.975020 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.974926    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:36:56.978825 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.978799    2583 generic.go:356] "Generic (PLEG): container finished" podID="005c2972-33cc-4d93-a617-b4542865e869" containerID="45c83dada7f678d968bbe5864cae21237d334b90da29713b0b718289148684e8" exitCode=2
Sep 17 19:36:56.978927 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.978868    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" event={"ID":"005c2972-33cc-4d93-a617-b4542865e869","Type":"ContainerDied","Data":"45c83dada7f678d968bbe5864cae21237d334b90da29713b0b718289148684e8"}
Sep 17 19:36:56.983724 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.983697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" event={"ID":"536aa7c2-08cf-4a18-b895-c4c8997acafb","Type":"ContainerStarted","Data":"4722365b5899cafcff26f0cbba152648bfd4fcbe0091626a3bb440d7d18e1917"}
Sep 17 19:36:56.983844 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.983766    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:36:56.983944 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.983895    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:36:56.984064 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.983948    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" event={"ID":"536aa7c2-08cf-4a18-b895-c4c8997acafb","Type":"ContainerStarted","Data":"0daeeb56010dc1f4e93d18532a3b3dd825eca368994a36e6d3398ea21641564b"}
Sep 17 19:36:56.984064 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.983979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" event={"ID":"536aa7c2-08cf-4a18-b895-c4c8997acafb","Type":"ContainerStarted","Data":"d2e936c3d3125b01c3b4c4d5644c0bd5ea0bdcb1db5767073c67f010c30ea7f1"}
Sep 17 19:36:56.985109 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.985083    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:36:56.985894 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.985874    2583 generic.go:356] "Generic (PLEG): container finished" podID="35869f6f-277a-422e-9cb7-d63e380e437e" containerID="4f5673279d2e079a81afe87c195a9a0be7ec51ebec21c2106b6e931bcb868a0a" exitCode=2
Sep 17 19:36:56.985986 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:56.985904    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" event={"ID":"35869f6f-277a-422e-9cb7-d63e380e437e","Type":"ContainerDied","Data":"4f5673279d2e079a81afe87c195a9a0be7ec51ebec21c2106b6e931bcb868a0a"}
Sep 17 19:36:57.001724 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:57.001681    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" podStartSLOduration=1.001666175 podStartE2EDuration="1.001666175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:36:57.000972819 +0000 UTC m=+1820.828831082" watchObservedRunningTime="2026-09-17 19:36:57.001666175 +0000 UTC m=+1820.829524422"
Sep 17 19:36:57.250556 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:57.250530    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"]
Sep 17 19:36:57.252416 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:36:57.252384    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b188c85_659e_4241_8c59_f2d68a54da68.slice/crio-91cc47033311e1b890797b23321326efef7a13241b56b560357b763dc87dee7c WatchSource:0}: Error finding container 91cc47033311e1b890797b23321326efef7a13241b56b560357b763dc87dee7c: Status 404 returned error can't find the container with id 91cc47033311e1b890797b23321326efef7a13241b56b560357b763dc87dee7c
Sep 17 19:36:57.340963 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:57.340934    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 17 19:36:57.341038 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:57.340990    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 17 19:36:57.346150 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:57.346125    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 19:36:57.346243 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:57.346164    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 19:36:57.992712 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:57.992672    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" event={"ID":"0b188c85-659e-4241-8c59-f2d68a54da68","Type":"ContainerStarted","Data":"c2cdcf4aa1c620b74bbfa24201fe91d006096881817bc27aee5bb0a790871ad0"}
Sep 17 19:36:57.993180 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:57.992754    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" event={"ID":"0b188c85-659e-4241-8c59-f2d68a54da68","Type":"ContainerStarted","Data":"6589b15eb958850f3b13411036e740ec434fa105990e09109b20a5b3d19c2f6b"}
Sep 17 19:36:57.993180 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:57.992783    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" event={"ID":"0b188c85-659e-4241-8c59-f2d68a54da68","Type":"ContainerStarted","Data":"91cc47033311e1b890797b23321326efef7a13241b56b560357b763dc87dee7c"}
Sep 17 19:36:57.993180 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:57.992939    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:36:57.993180 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:57.992992    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:36:57.993180 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:57.993011    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:36:57.994393 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:57.994367    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:36:58.014424 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:58.014367    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" podStartSLOduration=2.014348858 podStartE2EDuration="2.014348858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:36:58.013009552 +0000 UTC m=+1821.840867787" watchObservedRunningTime="2026-09-17 19:36:58.014348858 +0000 UTC m=+1821.842207094"
Sep 17 19:36:58.997230 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:58.997187    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:36:59.936704 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:59.936672    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:36:59.977294 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:36:59.977238    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:37:00.003783 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.003753    2583 generic.go:356] "Generic (PLEG): container finished" podID="005c2972-33cc-4d93-a617-b4542865e869" containerID="41bd1e0b64f26231b4d15f98079a4648a1f4c9aa4a36111f712dfef9bdd11cd2" exitCode=0
Sep 17 19:37:00.004198 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.003819    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"
Sep 17 19:37:00.004198 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.003837    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" event={"ID":"005c2972-33cc-4d93-a617-b4542865e869","Type":"ContainerDied","Data":"41bd1e0b64f26231b4d15f98079a4648a1f4c9aa4a36111f712dfef9bdd11cd2"}
Sep 17 19:37:00.004198 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.003917    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6" event={"ID":"005c2972-33cc-4d93-a617-b4542865e869","Type":"ContainerDied","Data":"c237fc90eeab9db9ceba8a9ae715f8b4f05a295be1fb5a69cdb09bb873d23641"}
Sep 17 19:37:00.004198 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.003933    2583 scope.go:117] "RemoveContainer" containerID="45c83dada7f678d968bbe5864cae21237d334b90da29713b0b718289148684e8"
Sep 17 19:37:00.005570 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.005550    2583 generic.go:356] "Generic (PLEG): container finished" podID="35869f6f-277a-422e-9cb7-d63e380e437e" containerID="01160fa5cac79e4d89b438def0e2d02a9df897548b8ca714ce07963881ffb6b3" exitCode=0
Sep 17 19:37:00.005665 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.005585    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" event={"ID":"35869f6f-277a-422e-9cb7-d63e380e437e","Type":"ContainerDied","Data":"01160fa5cac79e4d89b438def0e2d02a9df897548b8ca714ce07963881ffb6b3"}
Sep 17 19:37:00.005665 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.005595    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"
Sep 17 19:37:00.005665 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.005633    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j" event={"ID":"35869f6f-277a-422e-9cb7-d63e380e437e","Type":"ContainerDied","Data":"1d74f621aba381307ba906bcac29facba217cedc4e21b44aed2e61d62f982a20"}
Sep 17 19:37:00.013178 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.013158    2583 scope.go:117] "RemoveContainer" containerID="41bd1e0b64f26231b4d15f98079a4648a1f4c9aa4a36111f712dfef9bdd11cd2"
Sep 17 19:37:00.022430 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.022405    2583 scope.go:117] "RemoveContainer" containerID="45c83dada7f678d968bbe5864cae21237d334b90da29713b0b718289148684e8"
Sep 17 19:37:00.022815 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:37:00.022784    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c83dada7f678d968bbe5864cae21237d334b90da29713b0b718289148684e8\": container with ID starting with 45c83dada7f678d968bbe5864cae21237d334b90da29713b0b718289148684e8 not found: ID does not exist" containerID="45c83dada7f678d968bbe5864cae21237d334b90da29713b0b718289148684e8"
Sep 17 19:37:00.022932 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.022826    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c83dada7f678d968bbe5864cae21237d334b90da29713b0b718289148684e8"} err="failed to get container status \"45c83dada7f678d968bbe5864cae21237d334b90da29713b0b718289148684e8\": rpc error: code = NotFound desc = could not find container \"45c83dada7f678d968bbe5864cae21237d334b90da29713b0b718289148684e8\": container with ID starting with 45c83dada7f678d968bbe5864cae21237d334b90da29713b0b718289148684e8 not found: ID does not exist"
Sep 17 19:37:00.022932 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.022853    2583 scope.go:117] "RemoveContainer" containerID="41bd1e0b64f26231b4d15f98079a4648a1f4c9aa4a36111f712dfef9bdd11cd2"
Sep 17 19:37:00.023162 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:37:00.023146    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41bd1e0b64f26231b4d15f98079a4648a1f4c9aa4a36111f712dfef9bdd11cd2\": container with ID starting with 41bd1e0b64f26231b4d15f98079a4648a1f4c9aa4a36111f712dfef9bdd11cd2 not found: ID does not exist" containerID="41bd1e0b64f26231b4d15f98079a4648a1f4c9aa4a36111f712dfef9bdd11cd2"
Sep 17 19:37:00.023230 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.023166    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41bd1e0b64f26231b4d15f98079a4648a1f4c9aa4a36111f712dfef9bdd11cd2"} err="failed to get container status \"41bd1e0b64f26231b4d15f98079a4648a1f4c9aa4a36111f712dfef9bdd11cd2\": rpc error: code = NotFound desc = could not find container \"41bd1e0b64f26231b4d15f98079a4648a1f4c9aa4a36111f712dfef9bdd11cd2\": container with ID starting with 41bd1e0b64f26231b4d15f98079a4648a1f4c9aa4a36111f712dfef9bdd11cd2 not found: ID does not exist"
Sep 17 19:37:00.023230 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.023180    2583 scope.go:117] "RemoveContainer" containerID="4f5673279d2e079a81afe87c195a9a0be7ec51ebec21c2106b6e931bcb868a0a"
Sep 17 19:37:00.030387 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.030359    2583 scope.go:117] "RemoveContainer" containerID="01160fa5cac79e4d89b438def0e2d02a9df897548b8ca714ce07963881ffb6b3"
Sep 17 19:37:00.037720 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.037705    2583 scope.go:117] "RemoveContainer" containerID="4f5673279d2e079a81afe87c195a9a0be7ec51ebec21c2106b6e931bcb868a0a"
Sep 17 19:37:00.037970 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:37:00.037948    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f5673279d2e079a81afe87c195a9a0be7ec51ebec21c2106b6e931bcb868a0a\": container with ID starting with 4f5673279d2e079a81afe87c195a9a0be7ec51ebec21c2106b6e931bcb868a0a not found: ID does not exist" containerID="4f5673279d2e079a81afe87c195a9a0be7ec51ebec21c2106b6e931bcb868a0a"
Sep 17 19:37:00.038025 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.037977    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f5673279d2e079a81afe87c195a9a0be7ec51ebec21c2106b6e931bcb868a0a"} err="failed to get container status \"4f5673279d2e079a81afe87c195a9a0be7ec51ebec21c2106b6e931bcb868a0a\": rpc error: code = NotFound desc = could not find container \"4f5673279d2e079a81afe87c195a9a0be7ec51ebec21c2106b6e931bcb868a0a\": container with ID starting with 4f5673279d2e079a81afe87c195a9a0be7ec51ebec21c2106b6e931bcb868a0a not found: ID does not exist"
Sep 17 19:37:00.038025 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.037993    2583 scope.go:117] "RemoveContainer" containerID="01160fa5cac79e4d89b438def0e2d02a9df897548b8ca714ce07963881ffb6b3"
Sep 17 19:37:00.038230 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:37:00.038216    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01160fa5cac79e4d89b438def0e2d02a9df897548b8ca714ce07963881ffb6b3\": container with ID starting with 01160fa5cac79e4d89b438def0e2d02a9df897548b8ca714ce07963881ffb6b3 not found: ID does not exist" containerID="01160fa5cac79e4d89b438def0e2d02a9df897548b8ca714ce07963881ffb6b3"
Sep 17 19:37:00.038273 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.038233    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01160fa5cac79e4d89b438def0e2d02a9df897548b8ca714ce07963881ffb6b3"} err="failed to get container status \"01160fa5cac79e4d89b438def0e2d02a9df897548b8ca714ce07963881ffb6b3\": rpc error: code = NotFound desc = could not find container \"01160fa5cac79e4d89b438def0e2d02a9df897548b8ca714ce07963881ffb6b3\": container with ID starting with 01160fa5cac79e4d89b438def0e2d02a9df897548b8ca714ce07963881ffb6b3 not found: ID does not exist"
Sep 17 19:37:00.046502 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.046471    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35869f6f-277a-422e-9cb7-d63e380e437e-proxy-tls\") pod \"35869f6f-277a-422e-9cb7-d63e380e437e\" (UID: \"35869f6f-277a-422e-9cb7-d63e380e437e\") "
Sep 17 19:37:00.046604 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.046514    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9ddg\" (UniqueName: \"kubernetes.io/projected/005c2972-33cc-4d93-a617-b4542865e869-kube-api-access-d9ddg\") pod \"005c2972-33cc-4d93-a617-b4542865e869\" (UID: \"005c2972-33cc-4d93-a617-b4542865e869\") "
Sep 17 19:37:00.046604 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.046537    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35869f6f-277a-422e-9cb7-d63e380e437e-kube-rbac-proxy-sar-config\") pod \"35869f6f-277a-422e-9cb7-d63e380e437e\" (UID: \"35869f6f-277a-422e-9cb7-d63e380e437e\") "
Sep 17 19:37:00.046604 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.046554    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/005c2972-33cc-4d93-a617-b4542865e869-kube-rbac-proxy-sar-config\") pod \"005c2972-33cc-4d93-a617-b4542865e869\" (UID: \"005c2972-33cc-4d93-a617-b4542865e869\") "
Sep 17 19:37:00.046766 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.046727    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mrqs\" (UniqueName: \"kubernetes.io/projected/35869f6f-277a-422e-9cb7-d63e380e437e-kube-api-access-2mrqs\") pod \"35869f6f-277a-422e-9cb7-d63e380e437e\" (UID: \"35869f6f-277a-422e-9cb7-d63e380e437e\") "
Sep 17 19:37:00.046850 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.046831    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005c2972-33cc-4d93-a617-b4542865e869-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "005c2972-33cc-4d93-a617-b4542865e869" (UID: "005c2972-33cc-4d93-a617-b4542865e869"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:37:00.046906 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.046850    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/005c2972-33cc-4d93-a617-b4542865e869-proxy-tls\") pod \"005c2972-33cc-4d93-a617-b4542865e869\" (UID: \"005c2972-33cc-4d93-a617-b4542865e869\") "
Sep 17 19:37:00.046906 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.046869    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35869f6f-277a-422e-9cb7-d63e380e437e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "35869f6f-277a-422e-9cb7-d63e380e437e" (UID: "35869f6f-277a-422e-9cb7-d63e380e437e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:37:00.047127 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.047111    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35869f6f-277a-422e-9cb7-d63e380e437e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:37:00.047172 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.047133    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/005c2972-33cc-4d93-a617-b4542865e869-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:37:00.048705 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.048683    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/005c2972-33cc-4d93-a617-b4542865e869-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "005c2972-33cc-4d93-a617-b4542865e869" (UID: "005c2972-33cc-4d93-a617-b4542865e869"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:37:00.048705 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.048691    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35869f6f-277a-422e-9cb7-d63e380e437e-kube-api-access-2mrqs" (OuterVolumeSpecName: "kube-api-access-2mrqs") pod "35869f6f-277a-422e-9cb7-d63e380e437e" (UID: "35869f6f-277a-422e-9cb7-d63e380e437e"). InnerVolumeSpecName "kube-api-access-2mrqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:37:00.048955 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.048939    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35869f6f-277a-422e-9cb7-d63e380e437e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35869f6f-277a-422e-9cb7-d63e380e437e" (UID: "35869f6f-277a-422e-9cb7-d63e380e437e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:37:00.049000 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.048950    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005c2972-33cc-4d93-a617-b4542865e869-kube-api-access-d9ddg" (OuterVolumeSpecName: "kube-api-access-d9ddg") pod "005c2972-33cc-4d93-a617-b4542865e869" (UID: "005c2972-33cc-4d93-a617-b4542865e869"). InnerVolumeSpecName "kube-api-access-d9ddg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:37:00.148242 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.148195    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35869f6f-277a-422e-9cb7-d63e380e437e-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:37:00.148242 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.148236    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9ddg\" (UniqueName: \"kubernetes.io/projected/005c2972-33cc-4d93-a617-b4542865e869-kube-api-access-d9ddg\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:37:00.148242 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.148246    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mrqs\" (UniqueName: \"kubernetes.io/projected/35869f6f-277a-422e-9cb7-d63e380e437e-kube-api-access-2mrqs\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:37:00.148242 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.148255    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/005c2972-33cc-4d93-a617-b4542865e869-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:37:00.329209 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.329177    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"]
Sep 17 19:37:00.333139 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.333112    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-31ec7-predictor-67569bb8b8-x6x5j"]
Sep 17 19:37:00.343738 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.343712    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"]
Sep 17 19:37:00.348540 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.348518    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31ec7-predictor-58cf9cff8f-jt9p6"]
Sep 17 19:37:00.452567 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.452533    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="005c2972-33cc-4d93-a617-b4542865e869" path="/var/lib/kubelet/pods/005c2972-33cc-4d93-a617-b4542865e869/volumes"
Sep 17 19:37:00.452974 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:00.452960    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" path="/var/lib/kubelet/pods/35869f6f-277a-422e-9cb7-d63e380e437e/volumes"
Sep 17 19:37:02.998782 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:02.998754    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:37:02.999313 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:02.999284    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:37:04.002184 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:04.002149    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:37:04.002798 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:04.002769    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:37:13.000057 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:13.000009    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:37:14.003464 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:14.003410    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:37:22.999483 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:22.999423    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:37:24.002830 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:24.002786    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:37:32.999310 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:32.999269    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:37:34.003188 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:34.003147    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:37:42.999659 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:42.999616    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 19:37:44.002806 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:44.002764    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 19:37:53.000403 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:53.000376    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:37:54.004270 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:54.004235    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:37:57.091400 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.091364    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"]
Sep 17 19:37:57.091843 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.091640    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kserve-container" containerID="cri-o://13f084db19c537f41a8ee15bf7874d36b1414d75c65046cec6774d69e963a33b" gracePeriod=30
Sep 17 19:37:57.091843 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.091680    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kube-rbac-proxy" containerID="cri-o://63c3cce2d2ad037c5b7571e026d3c2f9b607f089e7bdf043042125c1e5f97c5f" gracePeriod=30
Sep 17 19:37:57.141902 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.141871    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"]
Sep 17 19:37:57.142104 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.142083    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kserve-container" containerID="cri-o://64a4f2295a5e6e5458a21d874b4f9cf0c8ae89f1cb0cccd2447886e44a4c9613" gracePeriod=30
Sep 17 19:37:57.142194 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.142136    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kube-rbac-proxy" containerID="cri-o://947aa6c88adfe998ec33219e479144d95cf1ce3c877aa901c2465ac390b3a41a" gracePeriod=30
Sep 17 19:37:57.216383 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.216346    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"]
Sep 17 19:37:57.216802 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.216787    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kserve-container"
Sep 17 19:37:57.216858 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.216804    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kserve-container"
Sep 17 19:37:57.216858 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.216817    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kube-rbac-proxy"
Sep 17 19:37:57.216858 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.216823    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kube-rbac-proxy"
Sep 17 19:37:57.216858 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.216835    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kserve-container"
Sep 17 19:37:57.216858 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.216840    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kserve-container"
Sep 17 19:37:57.216858 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.216847    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kube-rbac-proxy"
Sep 17 19:37:57.216858 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.216852    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kube-rbac-proxy"
Sep 17 19:37:57.217102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.216904    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kserve-container"
Sep 17 19:37:57.217102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.216913    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="005c2972-33cc-4d93-a617-b4542865e869" containerName="kube-rbac-proxy"
Sep 17 19:37:57.217102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.216920    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kserve-container"
Sep 17 19:37:57.217102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.216937    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="35869f6f-277a-422e-9cb7-d63e380e437e" containerName="kube-rbac-proxy"
Sep 17 19:37:57.219983 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.219968    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:37:57.222671 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.222651    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-26209-predictor-serving-cert\""
Sep 17 19:37:57.222776 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.222734    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-26209-kube-rbac-proxy-sar-config\""
Sep 17 19:37:57.231438 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.231417    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"]
Sep 17 19:37:57.245991 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.245966    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zrc5\" (UniqueName: \"kubernetes.io/projected/247126b2-16da-4af0-a78b-63cdd2565bf3-kube-api-access-5zrc5\") pod \"success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8\" (UID: \"247126b2-16da-4af0-a78b-63cdd2565bf3\") " pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:37:57.246100 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.246032    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247126b2-16da-4af0-a78b-63cdd2565bf3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8\" (UID: \"247126b2-16da-4af0-a78b-63cdd2565bf3\") " pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:37:57.246165 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.246103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247126b2-16da-4af0-a78b-63cdd2565bf3-proxy-tls\") pod \"success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8\" (UID: \"247126b2-16da-4af0-a78b-63cdd2565bf3\") " pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:37:57.265731 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.265703    2583 generic.go:356] "Generic (PLEG): container finished" podID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerID="63c3cce2d2ad037c5b7571e026d3c2f9b607f089e7bdf043042125c1e5f97c5f" exitCode=2
Sep 17 19:37:57.265859 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.265768    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" event={"ID":"f4168fdd-5d8b-4a40-a773-5ab359e5e344","Type":"ContainerDied","Data":"63c3cce2d2ad037c5b7571e026d3c2f9b607f089e7bdf043042125c1e5f97c5f"}
Sep 17 19:37:57.267785 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.267757    2583 generic.go:356] "Generic (PLEG): container finished" podID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerID="947aa6c88adfe998ec33219e479144d95cf1ce3c877aa901c2465ac390b3a41a" exitCode=2
Sep 17 19:37:57.267873 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.267820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" event={"ID":"38870954-f193-4fe8-84ea-5a6ff226c37e","Type":"ContainerDied","Data":"947aa6c88adfe998ec33219e479144d95cf1ce3c877aa901c2465ac390b3a41a"}
Sep 17 19:37:57.311233 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.311204    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"]
Sep 17 19:37:57.314631 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.314608    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:37:57.319078 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.319054    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-26209-kube-rbac-proxy-sar-config\""
Sep 17 19:37:57.319201 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.319097    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-26209-predictor-serving-cert\""
Sep 17 19:37:57.324977 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.324953    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"]
Sep 17 19:37:57.347344 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.347285    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960b3886-5a54-450d-8584-1e041c1f5c59-proxy-tls\") pod \"error-404-isvc-26209-predictor-59d644fd44-z2ffv\" (UID: \"960b3886-5a54-450d-8584-1e041c1f5c59\") " pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:37:57.347344 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.347313    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/960b3886-5a54-450d-8584-1e041c1f5c59-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-26209-predictor-59d644fd44-z2ffv\" (UID: \"960b3886-5a54-450d-8584-1e041c1f5c59\") " pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:37:57.347344 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.347329    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb5k6\" (UniqueName: \"kubernetes.io/projected/960b3886-5a54-450d-8584-1e041c1f5c59-kube-api-access-sb5k6\") pod \"error-404-isvc-26209-predictor-59d644fd44-z2ffv\" (UID: \"960b3886-5a54-450d-8584-1e041c1f5c59\") " pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:37:57.347561 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.347349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zrc5\" (UniqueName: \"kubernetes.io/projected/247126b2-16da-4af0-a78b-63cdd2565bf3-kube-api-access-5zrc5\") pod \"success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8\" (UID: \"247126b2-16da-4af0-a78b-63cdd2565bf3\") " pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:37:57.347561 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.347504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247126b2-16da-4af0-a78b-63cdd2565bf3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8\" (UID: \"247126b2-16da-4af0-a78b-63cdd2565bf3\") " pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:37:57.347666 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.347575    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247126b2-16da-4af0-a78b-63cdd2565bf3-proxy-tls\") pod \"success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8\" (UID: \"247126b2-16da-4af0-a78b-63cdd2565bf3\") " pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:37:57.347717 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:37:57.347669    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-26209-predictor-serving-cert: secret "success-200-isvc-26209-predictor-serving-cert" not found
Sep 17 19:37:57.347717 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:37:57.347716    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247126b2-16da-4af0-a78b-63cdd2565bf3-proxy-tls podName:247126b2-16da-4af0-a78b-63cdd2565bf3 nodeName:}" failed. No retries permitted until 2026-09-17 19:37:57.847703826 +0000 UTC m=+1881.675562038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/247126b2-16da-4af0-a78b-63cdd2565bf3-proxy-tls") pod "success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" (UID: "247126b2-16da-4af0-a78b-63cdd2565bf3") : secret "success-200-isvc-26209-predictor-serving-cert" not found
Sep 17 19:37:57.348136 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.348114    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247126b2-16da-4af0-a78b-63cdd2565bf3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8\" (UID: \"247126b2-16da-4af0-a78b-63cdd2565bf3\") " pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:37:57.355930 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.355907    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zrc5\" (UniqueName: \"kubernetes.io/projected/247126b2-16da-4af0-a78b-63cdd2565bf3-kube-api-access-5zrc5\") pod \"success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8\" (UID: \"247126b2-16da-4af0-a78b-63cdd2565bf3\") " pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:37:57.448163 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.448130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960b3886-5a54-450d-8584-1e041c1f5c59-proxy-tls\") pod \"error-404-isvc-26209-predictor-59d644fd44-z2ffv\" (UID: \"960b3886-5a54-450d-8584-1e041c1f5c59\") " pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:37:57.448163 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.448166    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/960b3886-5a54-450d-8584-1e041c1f5c59-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-26209-predictor-59d644fd44-z2ffv\" (UID: \"960b3886-5a54-450d-8584-1e041c1f5c59\") " pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:37:57.448404 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.448181    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb5k6\" (UniqueName: \"kubernetes.io/projected/960b3886-5a54-450d-8584-1e041c1f5c59-kube-api-access-sb5k6\") pod \"error-404-isvc-26209-predictor-59d644fd44-z2ffv\" (UID: \"960b3886-5a54-450d-8584-1e041c1f5c59\") " pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:37:57.448404 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:37:57.448286    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-26209-predictor-serving-cert: secret "error-404-isvc-26209-predictor-serving-cert" not found
Sep 17 19:37:57.448404 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:37:57.448356    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/960b3886-5a54-450d-8584-1e041c1f5c59-proxy-tls podName:960b3886-5a54-450d-8584-1e041c1f5c59 nodeName:}" failed. No retries permitted until 2026-09-17 19:37:57.948338556 +0000 UTC m=+1881.776196769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/960b3886-5a54-450d-8584-1e041c1f5c59-proxy-tls") pod "error-404-isvc-26209-predictor-59d644fd44-z2ffv" (UID: "960b3886-5a54-450d-8584-1e041c1f5c59") : secret "error-404-isvc-26209-predictor-serving-cert" not found
Sep 17 19:37:57.448795 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.448778    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/960b3886-5a54-450d-8584-1e041c1f5c59-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-26209-predictor-59d644fd44-z2ffv\" (UID: \"960b3886-5a54-450d-8584-1e041c1f5c59\") " pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:37:57.458361 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.458343    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb5k6\" (UniqueName: \"kubernetes.io/projected/960b3886-5a54-450d-8584-1e041c1f5c59-kube-api-access-sb5k6\") pod \"error-404-isvc-26209-predictor-59d644fd44-z2ffv\" (UID: \"960b3886-5a54-450d-8584-1e041c1f5c59\") " pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:37:57.850930 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.850890    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247126b2-16da-4af0-a78b-63cdd2565bf3-proxy-tls\") pod \"success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8\" (UID: \"247126b2-16da-4af0-a78b-63cdd2565bf3\") " pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:37:57.853243 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.853224    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247126b2-16da-4af0-a78b-63cdd2565bf3-proxy-tls\") pod \"success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8\" (UID: \"247126b2-16da-4af0-a78b-63cdd2565bf3\") " pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:37:57.951792 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.951755    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960b3886-5a54-450d-8584-1e041c1f5c59-proxy-tls\") pod \"error-404-isvc-26209-predictor-59d644fd44-z2ffv\" (UID: \"960b3886-5a54-450d-8584-1e041c1f5c59\") " pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:37:57.954073 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:57.954052    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960b3886-5a54-450d-8584-1e041c1f5c59-proxy-tls\") pod \"error-404-isvc-26209-predictor-59d644fd44-z2ffv\" (UID: \"960b3886-5a54-450d-8584-1e041c1f5c59\") " pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:37:58.129731 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:58.129643    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:37:58.225299 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:58.225263    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:37:58.399637 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:58.399606    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"]
Sep 17 19:37:58.401371 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:37:58.401344    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247126b2_16da_4af0_a78b_63cdd2565bf3.slice/crio-369576f8cdf38e49695cfb32759fd9d8836e84eb378d5062264f6f01abb9f02f WatchSource:0}: Error finding container 369576f8cdf38e49695cfb32759fd9d8836e84eb378d5062264f6f01abb9f02f: Status 404 returned error can't find the container with id 369576f8cdf38e49695cfb32759fd9d8836e84eb378d5062264f6f01abb9f02f
Sep 17 19:37:58.496494 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:58.496472    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"]
Sep 17 19:37:58.497812 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:37:58.497784    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960b3886_5a54_450d_8584_1e041c1f5c59.slice/crio-4714f5177d854c2c892fb4d203c952dffed1ae636b8adc4b6d0de2557c06db62 WatchSource:0}: Error finding container 4714f5177d854c2c892fb4d203c952dffed1ae636b8adc4b6d0de2557c06db62: Status 404 returned error can't find the container with id 4714f5177d854c2c892fb4d203c952dffed1ae636b8adc4b6d0de2557c06db62
Sep 17 19:37:59.286569 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.286532    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" event={"ID":"247126b2-16da-4af0-a78b-63cdd2565bf3","Type":"ContainerStarted","Data":"b9b71f91eba9abdece17358c1f0bc6eccc85e283f40d5978e551bcf33d743e5e"}
Sep 17 19:37:59.287052 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.286738    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:37:59.287052 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.286805    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:37:59.287052 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.286835    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" event={"ID":"247126b2-16da-4af0-a78b-63cdd2565bf3","Type":"ContainerStarted","Data":"c10e273cf03f2c8b4e90bcdc851a0538037fd0a3156426f508e0ec4678f7969c"}
Sep 17 19:37:59.287052 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.286872    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" event={"ID":"247126b2-16da-4af0-a78b-63cdd2565bf3","Type":"ContainerStarted","Data":"369576f8cdf38e49695cfb32759fd9d8836e84eb378d5062264f6f01abb9f02f"}
Sep 17 19:37:59.288013 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.287978    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:37:59.288420 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.288400    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" event={"ID":"960b3886-5a54-450d-8584-1e041c1f5c59","Type":"ContainerStarted","Data":"e719d6c90d388b91ec0f926f5e41ac6251c0f3d5eef93875d44bbc99832b4786"}
Sep 17 19:37:59.288525 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.288472    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" event={"ID":"960b3886-5a54-450d-8584-1e041c1f5c59","Type":"ContainerStarted","Data":"b6d9d95a38590815e22b543e050c3c00e5412d3a32e75cfd0013f2afaa971961"}
Sep 17 19:37:59.288525 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.288512    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:37:59.288632 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.288541    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" event={"ID":"960b3886-5a54-450d-8584-1e041c1f5c59","Type":"ContainerStarted","Data":"4714f5177d854c2c892fb4d203c952dffed1ae636b8adc4b6d0de2557c06db62"}
Sep 17 19:37:59.308259 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.308219    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" podStartSLOduration=2.3082068700000002 podStartE2EDuration="2.30820687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:37:59.305716813 +0000 UTC m=+1883.133575047" watchObservedRunningTime="2026-09-17 19:37:59.30820687 +0000 UTC m=+1883.136065103"
Sep 17 19:37:59.325924 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.325879    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" podStartSLOduration=2.325864008 podStartE2EDuration="2.325864008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:37:59.324681606 +0000 UTC m=+1883.152539840" watchObservedRunningTime="2026-09-17 19:37:59.325864008 +0000 UTC m=+1883.153722221"
Sep 17 19:37:59.650580 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.650486    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 19:37:59.650580 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.650481    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 17 19:37:59.656286 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.656264    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 19:37:59.656337 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:37:59.656262    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 19:38:00.293209 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:00.293172    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:38:00.293209 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:00.293172    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:38:00.294318 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:00.294293    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 19:38:00.748402 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:00.748378    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:38:00.875904 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:00.875816    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4168fdd-5d8b-4a40-a773-5ab359e5e344-proxy-tls\") pod \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\" (UID: \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\") "
Sep 17 19:38:00.875904 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:00.875858    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4168fdd-5d8b-4a40-a773-5ab359e5e344-kube-rbac-proxy-sar-config\") pod \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\" (UID: \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\") "
Sep 17 19:38:00.876099 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:00.875947    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxnnj\" (UniqueName: \"kubernetes.io/projected/f4168fdd-5d8b-4a40-a773-5ab359e5e344-kube-api-access-cxnnj\") pod \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\" (UID: \"f4168fdd-5d8b-4a40-a773-5ab359e5e344\") "
Sep 17 19:38:00.876330 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:00.876306    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4168fdd-5d8b-4a40-a773-5ab359e5e344-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f4168fdd-5d8b-4a40-a773-5ab359e5e344" (UID: "f4168fdd-5d8b-4a40-a773-5ab359e5e344"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:38:00.878030 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:00.878004    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4168fdd-5d8b-4a40-a773-5ab359e5e344-kube-api-access-cxnnj" (OuterVolumeSpecName: "kube-api-access-cxnnj") pod "f4168fdd-5d8b-4a40-a773-5ab359e5e344" (UID: "f4168fdd-5d8b-4a40-a773-5ab359e5e344"). InnerVolumeSpecName "kube-api-access-cxnnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:38:00.878115 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:00.878060    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4168fdd-5d8b-4a40-a773-5ab359e5e344-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4168fdd-5d8b-4a40-a773-5ab359e5e344" (UID: "f4168fdd-5d8b-4a40-a773-5ab359e5e344"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:38:00.977201 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:00.977163    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxnnj\" (UniqueName: \"kubernetes.io/projected/f4168fdd-5d8b-4a40-a773-5ab359e5e344-kube-api-access-cxnnj\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:38:00.977201 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:00.977189    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4168fdd-5d8b-4a40-a773-5ab359e5e344-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:38:00.977201 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:00.977199    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4168fdd-5d8b-4a40-a773-5ab359e5e344-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:38:01.081209 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.081186    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:38:01.178070 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.177980    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38870954-f193-4fe8-84ea-5a6ff226c37e-kube-rbac-proxy-sar-config\") pod \"38870954-f193-4fe8-84ea-5a6ff226c37e\" (UID: \"38870954-f193-4fe8-84ea-5a6ff226c37e\") "
Sep 17 19:38:01.178070 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.178028    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38870954-f193-4fe8-84ea-5a6ff226c37e-proxy-tls\") pod \"38870954-f193-4fe8-84ea-5a6ff226c37e\" (UID: \"38870954-f193-4fe8-84ea-5a6ff226c37e\") "
Sep 17 19:38:01.178314 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.178094    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljd9s\" (UniqueName: \"kubernetes.io/projected/38870954-f193-4fe8-84ea-5a6ff226c37e-kube-api-access-ljd9s\") pod \"38870954-f193-4fe8-84ea-5a6ff226c37e\" (UID: \"38870954-f193-4fe8-84ea-5a6ff226c37e\") "
Sep 17 19:38:01.178418 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.178392    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38870954-f193-4fe8-84ea-5a6ff226c37e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "38870954-f193-4fe8-84ea-5a6ff226c37e" (UID: "38870954-f193-4fe8-84ea-5a6ff226c37e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:38:01.180112 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.180093    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38870954-f193-4fe8-84ea-5a6ff226c37e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "38870954-f193-4fe8-84ea-5a6ff226c37e" (UID: "38870954-f193-4fe8-84ea-5a6ff226c37e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:38:01.180163 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.180119    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38870954-f193-4fe8-84ea-5a6ff226c37e-kube-api-access-ljd9s" (OuterVolumeSpecName: "kube-api-access-ljd9s") pod "38870954-f193-4fe8-84ea-5a6ff226c37e" (UID: "38870954-f193-4fe8-84ea-5a6ff226c37e"). InnerVolumeSpecName "kube-api-access-ljd9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:38:01.278634 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.278605    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38870954-f193-4fe8-84ea-5a6ff226c37e-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:38:01.278634 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.278629    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljd9s\" (UniqueName: \"kubernetes.io/projected/38870954-f193-4fe8-84ea-5a6ff226c37e-kube-api-access-ljd9s\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:38:01.278634 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.278638    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38870954-f193-4fe8-84ea-5a6ff226c37e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:38:01.299778 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.299743    2583 generic.go:356] "Generic (PLEG): container finished" podID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerID="13f084db19c537f41a8ee15bf7874d36b1414d75c65046cec6774d69e963a33b" exitCode=0
Sep 17 19:38:01.300150 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.299792    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"
Sep 17 19:38:01.300150 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.299825    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" event={"ID":"f4168fdd-5d8b-4a40-a773-5ab359e5e344","Type":"ContainerDied","Data":"13f084db19c537f41a8ee15bf7874d36b1414d75c65046cec6774d69e963a33b"}
Sep 17 19:38:01.300150 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.299888    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw" event={"ID":"f4168fdd-5d8b-4a40-a773-5ab359e5e344","Type":"ContainerDied","Data":"3cbc1e07c2b670e49b3fecd3c3e59a5f12fd75c09a9aef0f2949e78e90869130"}
Sep 17 19:38:01.300150 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.299926    2583 scope.go:117] "RemoveContainer" containerID="63c3cce2d2ad037c5b7571e026d3c2f9b607f089e7bdf043042125c1e5f97c5f"
Sep 17 19:38:01.301496 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.301473    2583 generic.go:356] "Generic (PLEG): container finished" podID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerID="64a4f2295a5e6e5458a21d874b4f9cf0c8ae89f1cb0cccd2447886e44a4c9613" exitCode=0
Sep 17 19:38:01.301597 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.301536    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"
Sep 17 19:38:01.301662 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.301587    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" event={"ID":"38870954-f193-4fe8-84ea-5a6ff226c37e","Type":"ContainerDied","Data":"64a4f2295a5e6e5458a21d874b4f9cf0c8ae89f1cb0cccd2447886e44a4c9613"}
Sep 17 19:38:01.301662 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.301645    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm" event={"ID":"38870954-f193-4fe8-84ea-5a6ff226c37e","Type":"ContainerDied","Data":"e9c20199b378ca2366a18e86c4f3acedb96846edcf8e30c06088eeebdf0420d4"}
Sep 17 19:38:01.302046 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.301993    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 19:38:01.308959 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.308940    2583 scope.go:117] "RemoveContainer" containerID="13f084db19c537f41a8ee15bf7874d36b1414d75c65046cec6774d69e963a33b"
Sep 17 19:38:01.317571 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.317545    2583 scope.go:117] "RemoveContainer" containerID="63c3cce2d2ad037c5b7571e026d3c2f9b607f089e7bdf043042125c1e5f97c5f"
Sep 17 19:38:01.317863 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:38:01.317843    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c3cce2d2ad037c5b7571e026d3c2f9b607f089e7bdf043042125c1e5f97c5f\": container with ID starting with 63c3cce2d2ad037c5b7571e026d3c2f9b607f089e7bdf043042125c1e5f97c5f not found: ID does not exist" containerID="63c3cce2d2ad037c5b7571e026d3c2f9b607f089e7bdf043042125c1e5f97c5f"
Sep 17 19:38:01.317926 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.317871    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c3cce2d2ad037c5b7571e026d3c2f9b607f089e7bdf043042125c1e5f97c5f"} err="failed to get container status \"63c3cce2d2ad037c5b7571e026d3c2f9b607f089e7bdf043042125c1e5f97c5f\": rpc error: code = NotFound desc = could not find container \"63c3cce2d2ad037c5b7571e026d3c2f9b607f089e7bdf043042125c1e5f97c5f\": container with ID starting with 63c3cce2d2ad037c5b7571e026d3c2f9b607f089e7bdf043042125c1e5f97c5f not found: ID does not exist"
Sep 17 19:38:01.317926 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.317887    2583 scope.go:117] "RemoveContainer" containerID="13f084db19c537f41a8ee15bf7874d36b1414d75c65046cec6774d69e963a33b"
Sep 17 19:38:01.318160 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:38:01.318137    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f084db19c537f41a8ee15bf7874d36b1414d75c65046cec6774d69e963a33b\": container with ID starting with 13f084db19c537f41a8ee15bf7874d36b1414d75c65046cec6774d69e963a33b not found: ID does not exist" containerID="13f084db19c537f41a8ee15bf7874d36b1414d75c65046cec6774d69e963a33b"
Sep 17 19:38:01.318216 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.318162    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f084db19c537f41a8ee15bf7874d36b1414d75c65046cec6774d69e963a33b"} err="failed to get container status \"13f084db19c537f41a8ee15bf7874d36b1414d75c65046cec6774d69e963a33b\": rpc error: code = NotFound desc = could not find container \"13f084db19c537f41a8ee15bf7874d36b1414d75c65046cec6774d69e963a33b\": container with ID starting with 13f084db19c537f41a8ee15bf7874d36b1414d75c65046cec6774d69e963a33b not found: ID does not exist"
Sep 17 19:38:01.318216 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.318174    2583 scope.go:117] "RemoveContainer" containerID="947aa6c88adfe998ec33219e479144d95cf1ce3c877aa901c2465ac390b3a41a"
Sep 17 19:38:01.326321 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.326297    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"]
Sep 17 19:38:01.326607 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.326595    2583 scope.go:117] "RemoveContainer" containerID="64a4f2295a5e6e5458a21d874b4f9cf0c8ae89f1cb0cccd2447886e44a4c9613"
Sep 17 19:38:01.331523 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.331502    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cedb0-predictor-57c4b47646-c9hsm"]
Sep 17 19:38:01.334632 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.334616    2583 scope.go:117] "RemoveContainer" containerID="947aa6c88adfe998ec33219e479144d95cf1ce3c877aa901c2465ac390b3a41a"
Sep 17 19:38:01.334897 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:38:01.334876    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947aa6c88adfe998ec33219e479144d95cf1ce3c877aa901c2465ac390b3a41a\": container with ID starting with 947aa6c88adfe998ec33219e479144d95cf1ce3c877aa901c2465ac390b3a41a not found: ID does not exist" containerID="947aa6c88adfe998ec33219e479144d95cf1ce3c877aa901c2465ac390b3a41a"
Sep 17 19:38:01.334979 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.334902    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947aa6c88adfe998ec33219e479144d95cf1ce3c877aa901c2465ac390b3a41a"} err="failed to get container status \"947aa6c88adfe998ec33219e479144d95cf1ce3c877aa901c2465ac390b3a41a\": rpc error: code = NotFound desc = could not find container \"947aa6c88adfe998ec33219e479144d95cf1ce3c877aa901c2465ac390b3a41a\": container with ID starting with 947aa6c88adfe998ec33219e479144d95cf1ce3c877aa901c2465ac390b3a41a not found: ID does not exist"
Sep 17 19:38:01.334979 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.334920    2583 scope.go:117] "RemoveContainer" containerID="64a4f2295a5e6e5458a21d874b4f9cf0c8ae89f1cb0cccd2447886e44a4c9613"
Sep 17 19:38:01.335165 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:38:01.335147    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64a4f2295a5e6e5458a21d874b4f9cf0c8ae89f1cb0cccd2447886e44a4c9613\": container with ID starting with 64a4f2295a5e6e5458a21d874b4f9cf0c8ae89f1cb0cccd2447886e44a4c9613 not found: ID does not exist" containerID="64a4f2295a5e6e5458a21d874b4f9cf0c8ae89f1cb0cccd2447886e44a4c9613"
Sep 17 19:38:01.335209 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.335171    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64a4f2295a5e6e5458a21d874b4f9cf0c8ae89f1cb0cccd2447886e44a4c9613"} err="failed to get container status \"64a4f2295a5e6e5458a21d874b4f9cf0c8ae89f1cb0cccd2447886e44a4c9613\": rpc error: code = NotFound desc = could not find container \"64a4f2295a5e6e5458a21d874b4f9cf0c8ae89f1cb0cccd2447886e44a4c9613\": container with ID starting with 64a4f2295a5e6e5458a21d874b4f9cf0c8ae89f1cb0cccd2447886e44a4c9613 not found: ID does not exist"
Sep 17 19:38:01.344338 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.344318    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"]
Sep 17 19:38:01.349636 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:01.349617    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cedb0-predictor-5d689f5f55-fs6pw"]
Sep 17 19:38:02.452842 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:02.452809    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" path="/var/lib/kubelet/pods/38870954-f193-4fe8-84ea-5a6ff226c37e/volumes"
Sep 17 19:38:02.453239 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:02.453226    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" path="/var/lib/kubelet/pods/f4168fdd-5d8b-4a40-a773-5ab359e5e344/volumes"
Sep 17 19:38:05.298417 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:05.298392    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:38:05.298891 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:05.298868    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:38:06.307005 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:06.306978    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:38:06.307760 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:06.307731    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 19:38:15.299675 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:15.299635    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:38:16.308639 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:16.308602    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 19:38:17.310906 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.310875    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"]
Sep 17 19:38:17.311294 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.311144    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kserve-container" containerID="cri-o://6589b15eb958850f3b13411036e740ec434fa105990e09109b20a5b3d19c2f6b" gracePeriod=30
Sep 17 19:38:17.311294 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.311195    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kube-rbac-proxy" containerID="cri-o://c2cdcf4aa1c620b74bbfa24201fe91d006096881817bc27aee5bb0a790871ad0" gracePeriod=30
Sep 17 19:38:17.388528 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.388499    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"]
Sep 17 19:38:17.388749 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.388725    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kserve-container" containerID="cri-o://0daeeb56010dc1f4e93d18532a3b3dd825eca368994a36e6d3398ea21641564b" gracePeriod=30
Sep 17 19:38:17.388924 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.388763    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kube-rbac-proxy" containerID="cri-o://4722365b5899cafcff26f0cbba152648bfd4fcbe0091626a3bb440d7d18e1917" gracePeriod=30
Sep 17 19:38:17.392327 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.392307    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"]
Sep 17 19:38:17.392847 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.392828    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kube-rbac-proxy"
Sep 17 19:38:17.392937 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.392849    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kube-rbac-proxy"
Sep 17 19:38:17.392937 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.392863    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kube-rbac-proxy"
Sep 17 19:38:17.392937 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.392872    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kube-rbac-proxy"
Sep 17 19:38:17.392937 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.392881    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kserve-container"
Sep 17 19:38:17.392937 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.392891    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kserve-container"
Sep 17 19:38:17.392937 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.392911    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kserve-container"
Sep 17 19:38:17.392937 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.392920    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kserve-container"
Sep 17 19:38:17.393279 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.393022    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kube-rbac-proxy"
Sep 17 19:38:17.393279 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.393034    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kube-rbac-proxy"
Sep 17 19:38:17.393279 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.393046    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4168fdd-5d8b-4a40-a773-5ab359e5e344" containerName="kserve-container"
Sep 17 19:38:17.393279 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.393055    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="38870954-f193-4fe8-84ea-5a6ff226c37e" containerName="kserve-container"
Sep 17 19:38:17.396304 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.396284    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:38:17.398400 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.398378    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-21196-predictor-serving-cert\""
Sep 17 19:38:17.398668 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.398655    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-21196-kube-rbac-proxy-sar-config\""
Sep 17 19:38:17.403004 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.402983    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"]
Sep 17 19:38:17.511273 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.511246    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"]
Sep 17 19:38:17.514788 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.514768    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:38:17.517694 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.517672    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-21196-predictor-serving-cert\""
Sep 17 19:38:17.517827 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.517680    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-21196-kube-rbac-proxy-sar-config\""
Sep 17 19:38:17.525066 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.525042    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"]
Sep 17 19:38:17.532755 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.532731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hswmp\" (UniqueName: \"kubernetes.io/projected/66961fd2-01c2-48d2-899c-5847a7b53b56-kube-api-access-hswmp\") pod \"success-200-isvc-21196-predictor-9d4bcdc65-rwthp\" (UID: \"66961fd2-01c2-48d2-899c-5847a7b53b56\") " pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:38:17.532860 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.532847    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66961fd2-01c2-48d2-899c-5847a7b53b56-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-21196-predictor-9d4bcdc65-rwthp\" (UID: \"66961fd2-01c2-48d2-899c-5847a7b53b56\") " pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:38:17.533242 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.533218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66961fd2-01c2-48d2-899c-5847a7b53b56-proxy-tls\") pod \"success-200-isvc-21196-predictor-9d4bcdc65-rwthp\" (UID: \"66961fd2-01c2-48d2-899c-5847a7b53b56\") " pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:38:17.634604 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.634504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66961fd2-01c2-48d2-899c-5847a7b53b56-proxy-tls\") pod \"success-200-isvc-21196-predictor-9d4bcdc65-rwthp\" (UID: \"66961fd2-01c2-48d2-899c-5847a7b53b56\") " pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:38:17.634604 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.634566    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8833583-eddf-4ca0-85e4-9ae83fa224d4-proxy-tls\") pod \"error-404-isvc-21196-predictor-69485fb6d8-6dd9l\" (UID: \"c8833583-eddf-4ca0-85e4-9ae83fa224d4\") " pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:38:17.634604 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.634594    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hswmp\" (UniqueName: \"kubernetes.io/projected/66961fd2-01c2-48d2-899c-5847a7b53b56-kube-api-access-hswmp\") pod \"success-200-isvc-21196-predictor-9d4bcdc65-rwthp\" (UID: \"66961fd2-01c2-48d2-899c-5847a7b53b56\") " pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:38:17.634821 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.634615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h5zc\" (UniqueName: \"kubernetes.io/projected/c8833583-eddf-4ca0-85e4-9ae83fa224d4-kube-api-access-2h5zc\") pod \"error-404-isvc-21196-predictor-69485fb6d8-6dd9l\" (UID: \"c8833583-eddf-4ca0-85e4-9ae83fa224d4\") " pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:38:17.634821 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.634718    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8833583-eddf-4ca0-85e4-9ae83fa224d4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-21196-predictor-69485fb6d8-6dd9l\" (UID: \"c8833583-eddf-4ca0-85e4-9ae83fa224d4\") " pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:38:17.634821 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.634755    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66961fd2-01c2-48d2-899c-5847a7b53b56-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-21196-predictor-9d4bcdc65-rwthp\" (UID: \"66961fd2-01c2-48d2-899c-5847a7b53b56\") " pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:38:17.635393 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.635371    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66961fd2-01c2-48d2-899c-5847a7b53b56-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-21196-predictor-9d4bcdc65-rwthp\" (UID: \"66961fd2-01c2-48d2-899c-5847a7b53b56\") " pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:38:17.636875 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.636858    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66961fd2-01c2-48d2-899c-5847a7b53b56-proxy-tls\") pod \"success-200-isvc-21196-predictor-9d4bcdc65-rwthp\" (UID: \"66961fd2-01c2-48d2-899c-5847a7b53b56\") " pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:38:17.643492 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.643472    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hswmp\" (UniqueName: \"kubernetes.io/projected/66961fd2-01c2-48d2-899c-5847a7b53b56-kube-api-access-hswmp\") pod \"success-200-isvc-21196-predictor-9d4bcdc65-rwthp\" (UID: \"66961fd2-01c2-48d2-899c-5847a7b53b56\") " pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:38:17.712280 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.712242    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:38:17.735717 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.735686    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8833583-eddf-4ca0-85e4-9ae83fa224d4-proxy-tls\") pod \"error-404-isvc-21196-predictor-69485fb6d8-6dd9l\" (UID: \"c8833583-eddf-4ca0-85e4-9ae83fa224d4\") " pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:38:17.735857 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.735729    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h5zc\" (UniqueName: \"kubernetes.io/projected/c8833583-eddf-4ca0-85e4-9ae83fa224d4-kube-api-access-2h5zc\") pod \"error-404-isvc-21196-predictor-69485fb6d8-6dd9l\" (UID: \"c8833583-eddf-4ca0-85e4-9ae83fa224d4\") " pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:38:17.735857 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.735776    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8833583-eddf-4ca0-85e4-9ae83fa224d4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-21196-predictor-69485fb6d8-6dd9l\" (UID: \"c8833583-eddf-4ca0-85e4-9ae83fa224d4\") " pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:38:17.736416 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.736393    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8833583-eddf-4ca0-85e4-9ae83fa224d4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-21196-predictor-69485fb6d8-6dd9l\" (UID: \"c8833583-eddf-4ca0-85e4-9ae83fa224d4\") " pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:38:17.738102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.738083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8833583-eddf-4ca0-85e4-9ae83fa224d4-proxy-tls\") pod \"error-404-isvc-21196-predictor-69485fb6d8-6dd9l\" (UID: \"c8833583-eddf-4ca0-85e4-9ae83fa224d4\") " pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:38:17.743936 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.743916    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h5zc\" (UniqueName: \"kubernetes.io/projected/c8833583-eddf-4ca0-85e4-9ae83fa224d4-kube-api-access-2h5zc\") pod \"error-404-isvc-21196-predictor-69485fb6d8-6dd9l\" (UID: \"c8833583-eddf-4ca0-85e4-9ae83fa224d4\") " pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:38:17.827792 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.827750    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:38:17.986238 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.986215    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"]
Sep 17 19:38:17.987901 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:38:17.987875    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66961fd2_01c2_48d2_899c_5847a7b53b56.slice/crio-034b1f023e106edb3cd2caadf316996ae9521946a7d952adcd53e3ce67779116 WatchSource:0}: Error finding container 034b1f023e106edb3cd2caadf316996ae9521946a7d952adcd53e3ce67779116: Status 404 returned error can't find the container with id 034b1f023e106edb3cd2caadf316996ae9521946a7d952adcd53e3ce67779116
Sep 17 19:38:17.993920 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:17.993890    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 17 19:38:18.099405 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.099383    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"]
Sep 17 19:38:18.101088 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:38:18.101067    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8833583_eddf_4ca0_85e4_9ae83fa224d4.slice/crio-747c581f4c21e9d4645d2e581008baa6dcd78989253b94f4f592950c93d22237 WatchSource:0}: Error finding container 747c581f4c21e9d4645d2e581008baa6dcd78989253b94f4f592950c93d22237: Status 404 returned error can't find the container with id 747c581f4c21e9d4645d2e581008baa6dcd78989253b94f4f592950c93d22237
Sep 17 19:38:18.390579 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.390548    2583 generic.go:356] "Generic (PLEG): container finished" podID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerID="4722365b5899cafcff26f0cbba152648bfd4fcbe0091626a3bb440d7d18e1917" exitCode=2
Sep 17 19:38:18.390974 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.390626    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" event={"ID":"536aa7c2-08cf-4a18-b895-c4c8997acafb","Type":"ContainerDied","Data":"4722365b5899cafcff26f0cbba152648bfd4fcbe0091626a3bb440d7d18e1917"}
Sep 17 19:38:18.392326 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.392304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" event={"ID":"c8833583-eddf-4ca0-85e4-9ae83fa224d4","Type":"ContainerStarted","Data":"a02bc8634525404e053d388b3f35963f7e72c1b0ed06c523308b37658c493771"}
Sep 17 19:38:18.392433 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.392347    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" event={"ID":"c8833583-eddf-4ca0-85e4-9ae83fa224d4","Type":"ContainerStarted","Data":"7e48f23bd579b0dc821e4cc4db186ba95f3e70009a96a6e7f54aca4f836296fc"}
Sep 17 19:38:18.392433 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.392371    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" event={"ID":"c8833583-eddf-4ca0-85e4-9ae83fa224d4","Type":"ContainerStarted","Data":"747c581f4c21e9d4645d2e581008baa6dcd78989253b94f4f592950c93d22237"}
Sep 17 19:38:18.392433 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.392400    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:38:18.392433 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.392422    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:38:18.394306 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.394257    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:38:18.394415 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.394282    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" event={"ID":"66961fd2-01c2-48d2-899c-5847a7b53b56","Type":"ContainerStarted","Data":"112de6b649e98a5be2d93f7228f69be94601d817dca0e972574caa7e897c1a49"}
Sep 17 19:38:18.394415 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.394383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" event={"ID":"66961fd2-01c2-48d2-899c-5847a7b53b56","Type":"ContainerStarted","Data":"e433bec1de9a2349c47f2b87697499e18d0e2fda523dc680ad0321f479e39971"}
Sep 17 19:38:18.394549 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.394417    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" event={"ID":"66961fd2-01c2-48d2-899c-5847a7b53b56","Type":"ContainerStarted","Data":"034b1f023e106edb3cd2caadf316996ae9521946a7d952adcd53e3ce67779116"}
Sep 17 19:38:18.394549 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.394473    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:38:18.394549 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.394501    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:38:18.395605 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.395579    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:38:18.396122 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.396101    2583 generic.go:356] "Generic (PLEG): container finished" podID="0b188c85-659e-4241-8c59-f2d68a54da68" containerID="c2cdcf4aa1c620b74bbfa24201fe91d006096881817bc27aee5bb0a790871ad0" exitCode=2
Sep 17 19:38:18.396185 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.396149    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" event={"ID":"0b188c85-659e-4241-8c59-f2d68a54da68","Type":"ContainerDied","Data":"c2cdcf4aa1c620b74bbfa24201fe91d006096881817bc27aee5bb0a790871ad0"}
Sep 17 19:38:18.416320 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.414290    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" podStartSLOduration=1.414273644 podStartE2EDuration="1.414273644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:38:18.412503251 +0000 UTC m=+1902.240361488" watchObservedRunningTime="2026-09-17 19:38:18.414273644 +0000 UTC m=+1902.242131878"
Sep 17 19:38:18.431894 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.431841    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" podStartSLOduration=1.431821931 podStartE2EDuration="1.431821931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:38:18.431099358 +0000 UTC m=+1902.258957592" watchObservedRunningTime="2026-09-17 19:38:18.431821931 +0000 UTC m=+1902.259680180"
Sep 17 19:38:18.998090 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:18.998052    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 17 19:38:19.402098 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:19.402052    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:38:19.402509 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:19.402113    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:38:21.312636 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.312613    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:38:21.422982 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.422711    2583 generic.go:356] "Generic (PLEG): container finished" podID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerID="0daeeb56010dc1f4e93d18532a3b3dd825eca368994a36e6d3398ea21641564b" exitCode=0
Sep 17 19:38:21.422982 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.422937    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" event={"ID":"536aa7c2-08cf-4a18-b895-c4c8997acafb","Type":"ContainerDied","Data":"0daeeb56010dc1f4e93d18532a3b3dd825eca368994a36e6d3398ea21641564b"}
Sep 17 19:38:21.427420 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.427210    2583 generic.go:356] "Generic (PLEG): container finished" podID="0b188c85-659e-4241-8c59-f2d68a54da68" containerID="6589b15eb958850f3b13411036e740ec434fa105990e09109b20a5b3d19c2f6b" exitCode=0
Sep 17 19:38:21.427420 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.427323    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"
Sep 17 19:38:21.427647 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.427427    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" event={"ID":"0b188c85-659e-4241-8c59-f2d68a54da68","Type":"ContainerDied","Data":"6589b15eb958850f3b13411036e740ec434fa105990e09109b20a5b3d19c2f6b"}
Sep 17 19:38:21.427647 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.427514    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w" event={"ID":"0b188c85-659e-4241-8c59-f2d68a54da68","Type":"ContainerDied","Data":"91cc47033311e1b890797b23321326efef7a13241b56b560357b763dc87dee7c"}
Sep 17 19:38:21.427647 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.427544    2583 scope.go:117] "RemoveContainer" containerID="c2cdcf4aa1c620b74bbfa24201fe91d006096881817bc27aee5bb0a790871ad0"
Sep 17 19:38:21.435872 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.435858    2583 scope.go:117] "RemoveContainer" containerID="6589b15eb958850f3b13411036e740ec434fa105990e09109b20a5b3d19c2f6b"
Sep 17 19:38:21.439792 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.439771    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:38:21.443429 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.443411    2583 scope.go:117] "RemoveContainer" containerID="c2cdcf4aa1c620b74bbfa24201fe91d006096881817bc27aee5bb0a790871ad0"
Sep 17 19:38:21.443738 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:38:21.443723    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2cdcf4aa1c620b74bbfa24201fe91d006096881817bc27aee5bb0a790871ad0\": container with ID starting with c2cdcf4aa1c620b74bbfa24201fe91d006096881817bc27aee5bb0a790871ad0 not found: ID does not exist" containerID="c2cdcf4aa1c620b74bbfa24201fe91d006096881817bc27aee5bb0a790871ad0"
Sep 17 19:38:21.443829 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.443747    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2cdcf4aa1c620b74bbfa24201fe91d006096881817bc27aee5bb0a790871ad0"} err="failed to get container status \"c2cdcf4aa1c620b74bbfa24201fe91d006096881817bc27aee5bb0a790871ad0\": rpc error: code = NotFound desc = could not find container \"c2cdcf4aa1c620b74bbfa24201fe91d006096881817bc27aee5bb0a790871ad0\": container with ID starting with c2cdcf4aa1c620b74bbfa24201fe91d006096881817bc27aee5bb0a790871ad0 not found: ID does not exist"
Sep 17 19:38:21.443829 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.443763    2583 scope.go:117] "RemoveContainer" containerID="6589b15eb958850f3b13411036e740ec434fa105990e09109b20a5b3d19c2f6b"
Sep 17 19:38:21.444001 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:38:21.443985    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6589b15eb958850f3b13411036e740ec434fa105990e09109b20a5b3d19c2f6b\": container with ID starting with 6589b15eb958850f3b13411036e740ec434fa105990e09109b20a5b3d19c2f6b not found: ID does not exist" containerID="6589b15eb958850f3b13411036e740ec434fa105990e09109b20a5b3d19c2f6b"
Sep 17 19:38:21.444043 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.444006    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6589b15eb958850f3b13411036e740ec434fa105990e09109b20a5b3d19c2f6b"} err="failed to get container status \"6589b15eb958850f3b13411036e740ec434fa105990e09109b20a5b3d19c2f6b\": rpc error: code = NotFound desc = could not find container \"6589b15eb958850f3b13411036e740ec434fa105990e09109b20a5b3d19c2f6b\": container with ID starting with 6589b15eb958850f3b13411036e740ec434fa105990e09109b20a5b3d19c2f6b not found: ID does not exist"
Sep 17 19:38:21.467732 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.467715    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b188c85-659e-4241-8c59-f2d68a54da68-kube-rbac-proxy-sar-config\") pod \"0b188c85-659e-4241-8c59-f2d68a54da68\" (UID: \"0b188c85-659e-4241-8c59-f2d68a54da68\") "
Sep 17 19:38:21.467843 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.467799    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b188c85-659e-4241-8c59-f2d68a54da68-proxy-tls\") pod \"0b188c85-659e-4241-8c59-f2d68a54da68\" (UID: \"0b188c85-659e-4241-8c59-f2d68a54da68\") "
Sep 17 19:38:21.467895 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.467841    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smk9b\" (UniqueName: \"kubernetes.io/projected/0b188c85-659e-4241-8c59-f2d68a54da68-kube-api-access-smk9b\") pod \"0b188c85-659e-4241-8c59-f2d68a54da68\" (UID: \"0b188c85-659e-4241-8c59-f2d68a54da68\") "
Sep 17 19:38:21.468130 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.468097    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b188c85-659e-4241-8c59-f2d68a54da68-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b188c85-659e-4241-8c59-f2d68a54da68" (UID: "0b188c85-659e-4241-8c59-f2d68a54da68"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:38:21.469799 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.469774    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b188c85-659e-4241-8c59-f2d68a54da68-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b188c85-659e-4241-8c59-f2d68a54da68" (UID: "0b188c85-659e-4241-8c59-f2d68a54da68"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:38:21.469874 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.469807    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b188c85-659e-4241-8c59-f2d68a54da68-kube-api-access-smk9b" (OuterVolumeSpecName: "kube-api-access-smk9b") pod "0b188c85-659e-4241-8c59-f2d68a54da68" (UID: "0b188c85-659e-4241-8c59-f2d68a54da68"). InnerVolumeSpecName "kube-api-access-smk9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:38:21.568373 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.568291    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536aa7c2-08cf-4a18-b895-c4c8997acafb-kube-rbac-proxy-sar-config\") pod \"536aa7c2-08cf-4a18-b895-c4c8997acafb\" (UID: \"536aa7c2-08cf-4a18-b895-c4c8997acafb\") "
Sep 17 19:38:21.568522 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.568376    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536aa7c2-08cf-4a18-b895-c4c8997acafb-proxy-tls\") pod \"536aa7c2-08cf-4a18-b895-c4c8997acafb\" (UID: \"536aa7c2-08cf-4a18-b895-c4c8997acafb\") "
Sep 17 19:38:21.568522 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.568402    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghj45\" (UniqueName: \"kubernetes.io/projected/536aa7c2-08cf-4a18-b895-c4c8997acafb-kube-api-access-ghj45\") pod \"536aa7c2-08cf-4a18-b895-c4c8997acafb\" (UID: \"536aa7c2-08cf-4a18-b895-c4c8997acafb\") "
Sep 17 19:38:21.568634 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.568621    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b188c85-659e-4241-8c59-f2d68a54da68-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:38:21.568672 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.568639    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b188c85-659e-4241-8c59-f2d68a54da68-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:38:21.568672 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.568657    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smk9b\" (UniqueName: \"kubernetes.io/projected/0b188c85-659e-4241-8c59-f2d68a54da68-kube-api-access-smk9b\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:38:21.568743 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.568684    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536aa7c2-08cf-4a18-b895-c4c8997acafb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "536aa7c2-08cf-4a18-b895-c4c8997acafb" (UID: "536aa7c2-08cf-4a18-b895-c4c8997acafb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:38:21.570482 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.570461    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536aa7c2-08cf-4a18-b895-c4c8997acafb-kube-api-access-ghj45" (OuterVolumeSpecName: "kube-api-access-ghj45") pod "536aa7c2-08cf-4a18-b895-c4c8997acafb" (UID: "536aa7c2-08cf-4a18-b895-c4c8997acafb"). InnerVolumeSpecName "kube-api-access-ghj45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:38:21.570556 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.570503    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536aa7c2-08cf-4a18-b895-c4c8997acafb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "536aa7c2-08cf-4a18-b895-c4c8997acafb" (UID: "536aa7c2-08cf-4a18-b895-c4c8997acafb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:38:21.669324 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.669278    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536aa7c2-08cf-4a18-b895-c4c8997acafb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:38:21.669324 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.669311    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536aa7c2-08cf-4a18-b895-c4c8997acafb-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:38:21.669324 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.669321    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghj45\" (UniqueName: \"kubernetes.io/projected/536aa7c2-08cf-4a18-b895-c4c8997acafb-kube-api-access-ghj45\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:38:21.749639 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.749608    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"]
Sep 17 19:38:21.752475 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:21.752429    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fe2a8-predictor-85fbfd5f9f-66l9w"]
Sep 17 19:38:22.435088 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:22.435062    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"
Sep 17 19:38:22.435532 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:22.435064    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj" event={"ID":"536aa7c2-08cf-4a18-b895-c4c8997acafb","Type":"ContainerDied","Data":"d2e936c3d3125b01c3b4c4d5644c0bd5ea0bdcb1db5767073c67f010c30ea7f1"}
Sep 17 19:38:22.435532 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:22.435194    2583 scope.go:117] "RemoveContainer" containerID="4722365b5899cafcff26f0cbba152648bfd4fcbe0091626a3bb440d7d18e1917"
Sep 17 19:38:22.444228 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:22.444210    2583 scope.go:117] "RemoveContainer" containerID="0daeeb56010dc1f4e93d18532a3b3dd825eca368994a36e6d3398ea21641564b"
Sep 17 19:38:22.452098 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:22.452074    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" path="/var/lib/kubelet/pods/0b188c85-659e-4241-8c59-f2d68a54da68/volumes"
Sep 17 19:38:22.458001 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:22.457982    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"]
Sep 17 19:38:22.462663 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:22.462645    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fe2a8-predictor-68889756d6-6mwlj"]
Sep 17 19:38:24.406061 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:24.406036    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:38:24.406523 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:24.406112    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:38:24.406523 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:24.406465    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:38:24.406598 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:24.406526    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:38:24.451380 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:24.451342    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" path="/var/lib/kubelet/pods/536aa7c2-08cf-4a18-b895-c4c8997acafb/volumes"
Sep 17 19:38:25.298890 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:25.298849    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:38:26.308433 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:26.308384    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 19:38:34.406477 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:34.406425    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:38:34.406848 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:34.406425    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:38:35.299396 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:35.299354    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:38:36.307671 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:36.307636    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 19:38:44.407271 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:44.407235    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:38:44.407271 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:44.407250    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:38:45.299567 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:45.299526    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 19:38:46.308027 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:46.307979    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 19:38:54.407086 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:54.407049    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:38:54.407499 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:54.407050    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:38:55.300117 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:55.300084    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:38:56.309267 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:38:56.309239    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:39:04.407070 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:39:04.407032    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:39:04.407475 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:39:04.407035    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:39:14.407427 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:39:14.407399    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:39:14.407902 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:39:14.407845    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:41:38.509715 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:41:38.509686    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:41:38.515564 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:41:38.515540    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:46:38.541142 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:46:38.541122    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:46:38.546018 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:46:38.545995    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:47:42.690574 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:42.690531    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"]
Sep 17 19:47:42.691166 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:42.690798    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kserve-container" containerID="cri-o://e433bec1de9a2349c47f2b87697499e18d0e2fda523dc680ad0321f479e39971" gracePeriod=30
Sep 17 19:47:42.691166 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:42.690820    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kube-rbac-proxy" containerID="cri-o://112de6b649e98a5be2d93f7228f69be94601d817dca0e972574caa7e897c1a49" gracePeriod=30
Sep 17 19:47:42.724734 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:42.724700    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"]
Sep 17 19:47:42.725035 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:42.724973    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kserve-container" containerID="cri-o://7e48f23bd579b0dc821e4cc4db186ba95f3e70009a96a6e7f54aca4f836296fc" gracePeriod=30
Sep 17 19:47:42.725035 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:42.724999    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kube-rbac-proxy" containerID="cri-o://a02bc8634525404e053d388b3f35963f7e72c1b0ed06c523308b37658c493771" gracePeriod=30
Sep 17 19:47:42.932987 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:42.932949    2583 generic.go:356] "Generic (PLEG): container finished" podID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerID="a02bc8634525404e053d388b3f35963f7e72c1b0ed06c523308b37658c493771" exitCode=2
Sep 17 19:47:42.933172 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:42.933021    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" event={"ID":"c8833583-eddf-4ca0-85e4-9ae83fa224d4","Type":"ContainerDied","Data":"a02bc8634525404e053d388b3f35963f7e72c1b0ed06c523308b37658c493771"}
Sep 17 19:47:42.934823 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:42.934799    2583 generic.go:356] "Generic (PLEG): container finished" podID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerID="112de6b649e98a5be2d93f7228f69be94601d817dca0e972574caa7e897c1a49" exitCode=2
Sep 17 19:47:42.934939 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:42.934865    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" event={"ID":"66961fd2-01c2-48d2-899c-5847a7b53b56","Type":"ContainerDied","Data":"112de6b649e98a5be2d93f7228f69be94601d817dca0e972574caa7e897c1a49"}
Sep 17 19:47:44.403157 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:44.403111    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 17 19:47:44.403565 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:44.403116    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 17 19:47:44.406489 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:44.406454    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 19:47:44.406607 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:44.406582    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 19:47:46.260239 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.260209    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:47:46.292138 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.292107    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:47:46.374970 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.374932    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66961fd2-01c2-48d2-899c-5847a7b53b56-kube-rbac-proxy-sar-config\") pod \"66961fd2-01c2-48d2-899c-5847a7b53b56\" (UID: \"66961fd2-01c2-48d2-899c-5847a7b53b56\") "
Sep 17 19:47:46.375145 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.375019    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hswmp\" (UniqueName: \"kubernetes.io/projected/66961fd2-01c2-48d2-899c-5847a7b53b56-kube-api-access-hswmp\") pod \"66961fd2-01c2-48d2-899c-5847a7b53b56\" (UID: \"66961fd2-01c2-48d2-899c-5847a7b53b56\") "
Sep 17 19:47:46.375145 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.375047    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66961fd2-01c2-48d2-899c-5847a7b53b56-proxy-tls\") pod \"66961fd2-01c2-48d2-899c-5847a7b53b56\" (UID: \"66961fd2-01c2-48d2-899c-5847a7b53b56\") "
Sep 17 19:47:46.375323 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.375298    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66961fd2-01c2-48d2-899c-5847a7b53b56-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66961fd2-01c2-48d2-899c-5847a7b53b56" (UID: "66961fd2-01c2-48d2-899c-5847a7b53b56"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:47:46.377122 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.377094    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66961fd2-01c2-48d2-899c-5847a7b53b56-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66961fd2-01c2-48d2-899c-5847a7b53b56" (UID: "66961fd2-01c2-48d2-899c-5847a7b53b56"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:47:46.377251 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.377212    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66961fd2-01c2-48d2-899c-5847a7b53b56-kube-api-access-hswmp" (OuterVolumeSpecName: "kube-api-access-hswmp") pod "66961fd2-01c2-48d2-899c-5847a7b53b56" (UID: "66961fd2-01c2-48d2-899c-5847a7b53b56"). InnerVolumeSpecName "kube-api-access-hswmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:47:46.475960 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.475925    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h5zc\" (UniqueName: \"kubernetes.io/projected/c8833583-eddf-4ca0-85e4-9ae83fa224d4-kube-api-access-2h5zc\") pod \"c8833583-eddf-4ca0-85e4-9ae83fa224d4\" (UID: \"c8833583-eddf-4ca0-85e4-9ae83fa224d4\") "
Sep 17 19:47:46.476136 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.476013    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8833583-eddf-4ca0-85e4-9ae83fa224d4-proxy-tls\") pod \"c8833583-eddf-4ca0-85e4-9ae83fa224d4\" (UID: \"c8833583-eddf-4ca0-85e4-9ae83fa224d4\") "
Sep 17 19:47:46.476136 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.476044    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8833583-eddf-4ca0-85e4-9ae83fa224d4-kube-rbac-proxy-sar-config\") pod \"c8833583-eddf-4ca0-85e4-9ae83fa224d4\" (UID: \"c8833583-eddf-4ca0-85e4-9ae83fa224d4\") "
Sep 17 19:47:46.476252 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.476192    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66961fd2-01c2-48d2-899c-5847a7b53b56-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:47:46.476252 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.476202    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hswmp\" (UniqueName: \"kubernetes.io/projected/66961fd2-01c2-48d2-899c-5847a7b53b56-kube-api-access-hswmp\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:47:46.476252 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.476211    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66961fd2-01c2-48d2-899c-5847a7b53b56-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:47:46.476459 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.476412    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8833583-eddf-4ca0-85e4-9ae83fa224d4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c8833583-eddf-4ca0-85e4-9ae83fa224d4" (UID: "c8833583-eddf-4ca0-85e4-9ae83fa224d4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:47:46.478144 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.478120    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8833583-eddf-4ca0-85e4-9ae83fa224d4-kube-api-access-2h5zc" (OuterVolumeSpecName: "kube-api-access-2h5zc") pod "c8833583-eddf-4ca0-85e4-9ae83fa224d4" (UID: "c8833583-eddf-4ca0-85e4-9ae83fa224d4"). InnerVolumeSpecName "kube-api-access-2h5zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:47:46.478144 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.478121    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8833583-eddf-4ca0-85e4-9ae83fa224d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8833583-eddf-4ca0-85e4-9ae83fa224d4" (UID: "c8833583-eddf-4ca0-85e4-9ae83fa224d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:47:46.577534 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.577498    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8833583-eddf-4ca0-85e4-9ae83fa224d4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:47:46.577534 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.577529    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h5zc\" (UniqueName: \"kubernetes.io/projected/c8833583-eddf-4ca0-85e4-9ae83fa224d4-kube-api-access-2h5zc\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:47:46.577534 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.577539    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8833583-eddf-4ca0-85e4-9ae83fa224d4-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:47:46.953946 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.953848    2583 generic.go:356] "Generic (PLEG): container finished" podID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerID="7e48f23bd579b0dc821e4cc4db186ba95f3e70009a96a6e7f54aca4f836296fc" exitCode=0
Sep 17 19:47:46.953946 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.953899    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"
Sep 17 19:47:46.953946 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.953927    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" event={"ID":"c8833583-eddf-4ca0-85e4-9ae83fa224d4","Type":"ContainerDied","Data":"7e48f23bd579b0dc821e4cc4db186ba95f3e70009a96a6e7f54aca4f836296fc"}
Sep 17 19:47:46.954220 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.953991    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l" event={"ID":"c8833583-eddf-4ca0-85e4-9ae83fa224d4","Type":"ContainerDied","Data":"747c581f4c21e9d4645d2e581008baa6dcd78989253b94f4f592950c93d22237"}
Sep 17 19:47:46.954220 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.954029    2583 scope.go:117] "RemoveContainer" containerID="a02bc8634525404e053d388b3f35963f7e72c1b0ed06c523308b37658c493771"
Sep 17 19:47:46.958171 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.957347    2583 generic.go:356] "Generic (PLEG): container finished" podID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerID="e433bec1de9a2349c47f2b87697499e18d0e2fda523dc680ad0321f479e39971" exitCode=0
Sep 17 19:47:46.958171 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.957655    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"
Sep 17 19:47:46.959061 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.959029    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" event={"ID":"66961fd2-01c2-48d2-899c-5847a7b53b56","Type":"ContainerDied","Data":"e433bec1de9a2349c47f2b87697499e18d0e2fda523dc680ad0321f479e39971"}
Sep 17 19:47:46.959194 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.959169    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp" event={"ID":"66961fd2-01c2-48d2-899c-5847a7b53b56","Type":"ContainerDied","Data":"034b1f023e106edb3cd2caadf316996ae9521946a7d952adcd53e3ce67779116"}
Sep 17 19:47:46.971113 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.971093    2583 scope.go:117] "RemoveContainer" containerID="7e48f23bd579b0dc821e4cc4db186ba95f3e70009a96a6e7f54aca4f836296fc"
Sep 17 19:47:46.979001 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.978980    2583 scope.go:117] "RemoveContainer" containerID="a02bc8634525404e053d388b3f35963f7e72c1b0ed06c523308b37658c493771"
Sep 17 19:47:46.979299 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:47:46.979280    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02bc8634525404e053d388b3f35963f7e72c1b0ed06c523308b37658c493771\": container with ID starting with a02bc8634525404e053d388b3f35963f7e72c1b0ed06c523308b37658c493771 not found: ID does not exist" containerID="a02bc8634525404e053d388b3f35963f7e72c1b0ed06c523308b37658c493771"
Sep 17 19:47:46.979351 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.979307    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02bc8634525404e053d388b3f35963f7e72c1b0ed06c523308b37658c493771"} err="failed to get container status \"a02bc8634525404e053d388b3f35963f7e72c1b0ed06c523308b37658c493771\": rpc error: code = NotFound desc = could not find container \"a02bc8634525404e053d388b3f35963f7e72c1b0ed06c523308b37658c493771\": container with ID starting with a02bc8634525404e053d388b3f35963f7e72c1b0ed06c523308b37658c493771 not found: ID does not exist"
Sep 17 19:47:46.979351 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.979324    2583 scope.go:117] "RemoveContainer" containerID="7e48f23bd579b0dc821e4cc4db186ba95f3e70009a96a6e7f54aca4f836296fc"
Sep 17 19:47:46.979605 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:47:46.979583    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e48f23bd579b0dc821e4cc4db186ba95f3e70009a96a6e7f54aca4f836296fc\": container with ID starting with 7e48f23bd579b0dc821e4cc4db186ba95f3e70009a96a6e7f54aca4f836296fc not found: ID does not exist" containerID="7e48f23bd579b0dc821e4cc4db186ba95f3e70009a96a6e7f54aca4f836296fc"
Sep 17 19:47:46.979654 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.979614    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e48f23bd579b0dc821e4cc4db186ba95f3e70009a96a6e7f54aca4f836296fc"} err="failed to get container status \"7e48f23bd579b0dc821e4cc4db186ba95f3e70009a96a6e7f54aca4f836296fc\": rpc error: code = NotFound desc = could not find container \"7e48f23bd579b0dc821e4cc4db186ba95f3e70009a96a6e7f54aca4f836296fc\": container with ID starting with 7e48f23bd579b0dc821e4cc4db186ba95f3e70009a96a6e7f54aca4f836296fc not found: ID does not exist"
Sep 17 19:47:46.979654 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.979635    2583 scope.go:117] "RemoveContainer" containerID="112de6b649e98a5be2d93f7228f69be94601d817dca0e972574caa7e897c1a49"
Sep 17 19:47:46.987505 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.987483    2583 scope.go:117] "RemoveContainer" containerID="e433bec1de9a2349c47f2b87697499e18d0e2fda523dc680ad0321f479e39971"
Sep 17 19:47:46.995085 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.995064    2583 scope.go:117] "RemoveContainer" containerID="112de6b649e98a5be2d93f7228f69be94601d817dca0e972574caa7e897c1a49"
Sep 17 19:47:46.995395 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:47:46.995375    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"112de6b649e98a5be2d93f7228f69be94601d817dca0e972574caa7e897c1a49\": container with ID starting with 112de6b649e98a5be2d93f7228f69be94601d817dca0e972574caa7e897c1a49 not found: ID does not exist" containerID="112de6b649e98a5be2d93f7228f69be94601d817dca0e972574caa7e897c1a49"
Sep 17 19:47:46.995481 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.995408    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"112de6b649e98a5be2d93f7228f69be94601d817dca0e972574caa7e897c1a49"} err="failed to get container status \"112de6b649e98a5be2d93f7228f69be94601d817dca0e972574caa7e897c1a49\": rpc error: code = NotFound desc = could not find container \"112de6b649e98a5be2d93f7228f69be94601d817dca0e972574caa7e897c1a49\": container with ID starting with 112de6b649e98a5be2d93f7228f69be94601d817dca0e972574caa7e897c1a49 not found: ID does not exist"
Sep 17 19:47:46.995481 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.995434    2583 scope.go:117] "RemoveContainer" containerID="e433bec1de9a2349c47f2b87697499e18d0e2fda523dc680ad0321f479e39971"
Sep 17 19:47:46.995735 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:47:46.995717    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e433bec1de9a2349c47f2b87697499e18d0e2fda523dc680ad0321f479e39971\": container with ID starting with e433bec1de9a2349c47f2b87697499e18d0e2fda523dc680ad0321f479e39971 not found: ID does not exist" containerID="e433bec1de9a2349c47f2b87697499e18d0e2fda523dc680ad0321f479e39971"
Sep 17 19:47:46.995776 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.995744    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e433bec1de9a2349c47f2b87697499e18d0e2fda523dc680ad0321f479e39971"} err="failed to get container status \"e433bec1de9a2349c47f2b87697499e18d0e2fda523dc680ad0321f479e39971\": rpc error: code = NotFound desc = could not find container \"e433bec1de9a2349c47f2b87697499e18d0e2fda523dc680ad0321f479e39971\": container with ID starting with e433bec1de9a2349c47f2b87697499e18d0e2fda523dc680ad0321f479e39971 not found: ID does not exist"
Sep 17 19:47:46.999919 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:46.999893    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"]
Sep 17 19:47:47.009164 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:47.009133    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21196-predictor-69485fb6d8-6dd9l"]
Sep 17 19:47:47.024306 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:47.024278    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"]
Sep 17 19:47:47.038193 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:47.038155    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-21196-predictor-9d4bcdc65-rwthp"]
Sep 17 19:47:48.453163 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:48.453126    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" path="/var/lib/kubelet/pods/66961fd2-01c2-48d2-899c-5847a7b53b56/volumes"
Sep 17 19:47:48.453575 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:47:48.453561    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" path="/var/lib/kubelet/pods/c8833583-eddf-4ca0-85e4-9ae83fa224d4/volumes"
Sep 17 19:51:38.567663 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:51:38.567543    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:51:38.574055 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:51:38.574034    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:55:27.467663 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:27.467623    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"]
Sep 17 19:55:27.468189 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:27.467864    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kserve-container" containerID="cri-o://c10e273cf03f2c8b4e90bcdc851a0538037fd0a3156426f508e0ec4678f7969c" gracePeriod=30
Sep 17 19:55:27.468189 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:27.467905    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kube-rbac-proxy" containerID="cri-o://b9b71f91eba9abdece17358c1f0bc6eccc85e283f40d5978e551bcf33d743e5e" gracePeriod=30
Sep 17 19:55:27.502881 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:27.502841    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"]
Sep 17 19:55:27.503176 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:27.503124    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kserve-container" containerID="cri-o://b6d9d95a38590815e22b543e050c3c00e5412d3a32e75cfd0013f2afaa971961" gracePeriod=30
Sep 17 19:55:27.503306 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:27.503189    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kube-rbac-proxy" containerID="cri-o://e719d6c90d388b91ec0f926f5e41ac6251c0f3d5eef93875d44bbc99832b4786" gracePeriod=30
Sep 17 19:55:28.019661 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:28.019619    2583 generic.go:356] "Generic (PLEG): container finished" podID="960b3886-5a54-450d-8584-1e041c1f5c59" containerID="e719d6c90d388b91ec0f926f5e41ac6251c0f3d5eef93875d44bbc99832b4786" exitCode=2
Sep 17 19:55:28.019843 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:28.019685    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" event={"ID":"960b3886-5a54-450d-8584-1e041c1f5c59","Type":"ContainerDied","Data":"e719d6c90d388b91ec0f926f5e41ac6251c0f3d5eef93875d44bbc99832b4786"}
Sep 17 19:55:28.021706 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:28.021680    2583 generic.go:356] "Generic (PLEG): container finished" podID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerID="b9b71f91eba9abdece17358c1f0bc6eccc85e283f40d5978e551bcf33d743e5e" exitCode=2
Sep 17 19:55:28.021857 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:28.021711    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" event={"ID":"247126b2-16da-4af0-a78b-63cdd2565bf3","Type":"ContainerDied","Data":"b9b71f91eba9abdece17358c1f0bc6eccc85e283f40d5978e551bcf33d743e5e"}
Sep 17 19:55:30.293741 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:30.293698    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 17 19:55:31.226228 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.226200    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:55:31.344616 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.344578    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247126b2-16da-4af0-a78b-63cdd2565bf3-proxy-tls\") pod \"247126b2-16da-4af0-a78b-63cdd2565bf3\" (UID: \"247126b2-16da-4af0-a78b-63cdd2565bf3\") "
Sep 17 19:55:31.345081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.344659    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247126b2-16da-4af0-a78b-63cdd2565bf3-kube-rbac-proxy-sar-config\") pod \"247126b2-16da-4af0-a78b-63cdd2565bf3\" (UID: \"247126b2-16da-4af0-a78b-63cdd2565bf3\") "
Sep 17 19:55:31.345081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.344704    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zrc5\" (UniqueName: \"kubernetes.io/projected/247126b2-16da-4af0-a78b-63cdd2565bf3-kube-api-access-5zrc5\") pod \"247126b2-16da-4af0-a78b-63cdd2565bf3\" (UID: \"247126b2-16da-4af0-a78b-63cdd2565bf3\") "
Sep 17 19:55:31.345081 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.345047    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247126b2-16da-4af0-a78b-63cdd2565bf3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "247126b2-16da-4af0-a78b-63cdd2565bf3" (UID: "247126b2-16da-4af0-a78b-63cdd2565bf3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:55:31.346704 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.346680    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/247126b2-16da-4af0-a78b-63cdd2565bf3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "247126b2-16da-4af0-a78b-63cdd2565bf3" (UID: "247126b2-16da-4af0-a78b-63cdd2565bf3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:55:31.346808 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.346784    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247126b2-16da-4af0-a78b-63cdd2565bf3-kube-api-access-5zrc5" (OuterVolumeSpecName: "kube-api-access-5zrc5") pod "247126b2-16da-4af0-a78b-63cdd2565bf3" (UID: "247126b2-16da-4af0-a78b-63cdd2565bf3"). InnerVolumeSpecName "kube-api-access-5zrc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:55:31.352760 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.352736    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:55:31.445548 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.445518    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960b3886-5a54-450d-8584-1e041c1f5c59-proxy-tls\") pod \"960b3886-5a54-450d-8584-1e041c1f5c59\" (UID: \"960b3886-5a54-450d-8584-1e041c1f5c59\") "
Sep 17 19:55:31.445730 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.445596    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/960b3886-5a54-450d-8584-1e041c1f5c59-kube-rbac-proxy-sar-config\") pod \"960b3886-5a54-450d-8584-1e041c1f5c59\" (UID: \"960b3886-5a54-450d-8584-1e041c1f5c59\") "
Sep 17 19:55:31.445730 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.445627    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb5k6\" (UniqueName: \"kubernetes.io/projected/960b3886-5a54-450d-8584-1e041c1f5c59-kube-api-access-sb5k6\") pod \"960b3886-5a54-450d-8584-1e041c1f5c59\" (UID: \"960b3886-5a54-450d-8584-1e041c1f5c59\") "
Sep 17 19:55:31.445804 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.445771    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/247126b2-16da-4af0-a78b-63cdd2565bf3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:55:31.445804 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.445788    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zrc5\" (UniqueName: \"kubernetes.io/projected/247126b2-16da-4af0-a78b-63cdd2565bf3-kube-api-access-5zrc5\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:55:31.445877 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.445802    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/247126b2-16da-4af0-a78b-63cdd2565bf3-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:55:31.445956 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.445933    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960b3886-5a54-450d-8584-1e041c1f5c59-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "960b3886-5a54-450d-8584-1e041c1f5c59" (UID: "960b3886-5a54-450d-8584-1e041c1f5c59"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:55:31.447677 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.447649    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/960b3886-5a54-450d-8584-1e041c1f5c59-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "960b3886-5a54-450d-8584-1e041c1f5c59" (UID: "960b3886-5a54-450d-8584-1e041c1f5c59"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:55:31.447778 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.447720    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960b3886-5a54-450d-8584-1e041c1f5c59-kube-api-access-sb5k6" (OuterVolumeSpecName: "kube-api-access-sb5k6") pod "960b3886-5a54-450d-8584-1e041c1f5c59" (UID: "960b3886-5a54-450d-8584-1e041c1f5c59"). InnerVolumeSpecName "kube-api-access-sb5k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:55:31.546955 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.546911    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/960b3886-5a54-450d-8584-1e041c1f5c59-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:55:31.546955 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.546950    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb5k6\" (UniqueName: \"kubernetes.io/projected/960b3886-5a54-450d-8584-1e041c1f5c59-kube-api-access-sb5k6\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:55:31.547172 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:31.546966    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960b3886-5a54-450d-8584-1e041c1f5c59-proxy-tls\") on node \"ip-10-0-128-177.ec2.internal\" DevicePath \"\""
Sep 17 19:55:32.042260 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.042233    2583 generic.go:356] "Generic (PLEG): container finished" podID="960b3886-5a54-450d-8584-1e041c1f5c59" containerID="b6d9d95a38590815e22b543e050c3c00e5412d3a32e75cfd0013f2afaa971961" exitCode=0
Sep 17 19:55:32.042431 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.042290    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"
Sep 17 19:55:32.042431 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.042320    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" event={"ID":"960b3886-5a54-450d-8584-1e041c1f5c59","Type":"ContainerDied","Data":"b6d9d95a38590815e22b543e050c3c00e5412d3a32e75cfd0013f2afaa971961"}
Sep 17 19:55:32.042431 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.042397    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" event={"ID":"960b3886-5a54-450d-8584-1e041c1f5c59","Type":"ContainerDied","Data":"4714f5177d854c2c892fb4d203c952dffed1ae636b8adc4b6d0de2557c06db62"}
Sep 17 19:55:32.042625 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.042435    2583 scope.go:117] "RemoveContainer" containerID="e719d6c90d388b91ec0f926f5e41ac6251c0f3d5eef93875d44bbc99832b4786"
Sep 17 19:55:32.044091 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.044069    2583 generic.go:356] "Generic (PLEG): container finished" podID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerID="c10e273cf03f2c8b4e90bcdc851a0538037fd0a3156426f508e0ec4678f7969c" exitCode=0
Sep 17 19:55:32.044194 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.044181    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" event={"ID":"247126b2-16da-4af0-a78b-63cdd2565bf3","Type":"ContainerDied","Data":"c10e273cf03f2c8b4e90bcdc851a0538037fd0a3156426f508e0ec4678f7969c"}
Sep 17 19:55:32.044259 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.044200    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"
Sep 17 19:55:32.044259 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.044235    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8" event={"ID":"247126b2-16da-4af0-a78b-63cdd2565bf3","Type":"ContainerDied","Data":"369576f8cdf38e49695cfb32759fd9d8836e84eb378d5062264f6f01abb9f02f"}
Sep 17 19:55:32.052290 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.052265    2583 scope.go:117] "RemoveContainer" containerID="b6d9d95a38590815e22b543e050c3c00e5412d3a32e75cfd0013f2afaa971961"
Sep 17 19:55:32.059955 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.059937    2583 scope.go:117] "RemoveContainer" containerID="e719d6c90d388b91ec0f926f5e41ac6251c0f3d5eef93875d44bbc99832b4786"
Sep 17 19:55:32.060227 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:55:32.060211    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e719d6c90d388b91ec0f926f5e41ac6251c0f3d5eef93875d44bbc99832b4786\": container with ID starting with e719d6c90d388b91ec0f926f5e41ac6251c0f3d5eef93875d44bbc99832b4786 not found: ID does not exist" containerID="e719d6c90d388b91ec0f926f5e41ac6251c0f3d5eef93875d44bbc99832b4786"
Sep 17 19:55:32.060273 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.060235    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e719d6c90d388b91ec0f926f5e41ac6251c0f3d5eef93875d44bbc99832b4786"} err="failed to get container status \"e719d6c90d388b91ec0f926f5e41ac6251c0f3d5eef93875d44bbc99832b4786\": rpc error: code = NotFound desc = could not find container \"e719d6c90d388b91ec0f926f5e41ac6251c0f3d5eef93875d44bbc99832b4786\": container with ID starting with e719d6c90d388b91ec0f926f5e41ac6251c0f3d5eef93875d44bbc99832b4786 not found: ID does not exist"
Sep 17 19:55:32.060273 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.060250    2583 scope.go:117] "RemoveContainer" containerID="b6d9d95a38590815e22b543e050c3c00e5412d3a32e75cfd0013f2afaa971961"
Sep 17 19:55:32.060475 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:55:32.060458    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6d9d95a38590815e22b543e050c3c00e5412d3a32e75cfd0013f2afaa971961\": container with ID starting with b6d9d95a38590815e22b543e050c3c00e5412d3a32e75cfd0013f2afaa971961 not found: ID does not exist" containerID="b6d9d95a38590815e22b543e050c3c00e5412d3a32e75cfd0013f2afaa971961"
Sep 17 19:55:32.060523 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.060481    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d9d95a38590815e22b543e050c3c00e5412d3a32e75cfd0013f2afaa971961"} err="failed to get container status \"b6d9d95a38590815e22b543e050c3c00e5412d3a32e75cfd0013f2afaa971961\": rpc error: code = NotFound desc = could not find container \"b6d9d95a38590815e22b543e050c3c00e5412d3a32e75cfd0013f2afaa971961\": container with ID starting with b6d9d95a38590815e22b543e050c3c00e5412d3a32e75cfd0013f2afaa971961 not found: ID does not exist"
Sep 17 19:55:32.060523 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.060496    2583 scope.go:117] "RemoveContainer" containerID="b9b71f91eba9abdece17358c1f0bc6eccc85e283f40d5978e551bcf33d743e5e"
Sep 17 19:55:32.067682 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.067658    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"]
Sep 17 19:55:32.067946 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.067927    2583 scope.go:117] "RemoveContainer" containerID="c10e273cf03f2c8b4e90bcdc851a0538037fd0a3156426f508e0ec4678f7969c"
Sep 17 19:55:32.071154 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.071135    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-26209-predictor-cdfb4dcd5-x4tz8"]
Sep 17 19:55:32.075553 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.075534    2583 scope.go:117] "RemoveContainer" containerID="b9b71f91eba9abdece17358c1f0bc6eccc85e283f40d5978e551bcf33d743e5e"
Sep 17 19:55:32.075844 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:55:32.075826    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b71f91eba9abdece17358c1f0bc6eccc85e283f40d5978e551bcf33d743e5e\": container with ID starting with b9b71f91eba9abdece17358c1f0bc6eccc85e283f40d5978e551bcf33d743e5e not found: ID does not exist" containerID="b9b71f91eba9abdece17358c1f0bc6eccc85e283f40d5978e551bcf33d743e5e"
Sep 17 19:55:32.075895 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.075855    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b71f91eba9abdece17358c1f0bc6eccc85e283f40d5978e551bcf33d743e5e"} err="failed to get container status \"b9b71f91eba9abdece17358c1f0bc6eccc85e283f40d5978e551bcf33d743e5e\": rpc error: code = NotFound desc = could not find container \"b9b71f91eba9abdece17358c1f0bc6eccc85e283f40d5978e551bcf33d743e5e\": container with ID starting with b9b71f91eba9abdece17358c1f0bc6eccc85e283f40d5978e551bcf33d743e5e not found: ID does not exist"
Sep 17 19:55:32.075895 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.075874    2583 scope.go:117] "RemoveContainer" containerID="c10e273cf03f2c8b4e90bcdc851a0538037fd0a3156426f508e0ec4678f7969c"
Sep 17 19:55:32.076120 ip-10-0-128-177 kubenswrapper[2583]: E0917 19:55:32.076103    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c10e273cf03f2c8b4e90bcdc851a0538037fd0a3156426f508e0ec4678f7969c\": container with ID starting with c10e273cf03f2c8b4e90bcdc851a0538037fd0a3156426f508e0ec4678f7969c not found: ID does not exist" containerID="c10e273cf03f2c8b4e90bcdc851a0538037fd0a3156426f508e0ec4678f7969c"
Sep 17 19:55:32.076161 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.076127    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c10e273cf03f2c8b4e90bcdc851a0538037fd0a3156426f508e0ec4678f7969c"} err="failed to get container status \"c10e273cf03f2c8b4e90bcdc851a0538037fd0a3156426f508e0ec4678f7969c\": rpc error: code = NotFound desc = could not find container \"c10e273cf03f2c8b4e90bcdc851a0538037fd0a3156426f508e0ec4678f7969c\": container with ID starting with c10e273cf03f2c8b4e90bcdc851a0538037fd0a3156426f508e0ec4678f7969c not found: ID does not exist"
Sep 17 19:55:32.082329 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.082309    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"]
Sep 17 19:55:32.084347 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.084328    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv"]
Sep 17 19:55:32.303149 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.303103    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-26209-predictor-59d644fd44-z2ffv" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 19:55:32.452622 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.452591    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" path="/var/lib/kubelet/pods/247126b2-16da-4af0-a78b-63cdd2565bf3/volumes"
Sep 17 19:55:32.453026 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:55:32.453013    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" path="/var/lib/kubelet/pods/960b3886-5a54-450d-8584-1e041c1f5c59/volumes"
Sep 17 19:56:00.005599 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.005564    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9dgbs/must-gather-8qzdq"]
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.005945    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.005960    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.005975    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kserve-container"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.005981    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kserve-container"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.005988    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kserve-container"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.005995    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kserve-container"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006005    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kserve-container"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006011    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kserve-container"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006018    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006024    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006036    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006044    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006050    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006056    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006061    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006068    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006073    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006079    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006086    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kserve-container"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006091    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kserve-container"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006101    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kserve-container"
Sep 17 19:56:00.006102 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006107    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kserve-container"
Sep 17 19:56:00.006812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006118    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kserve-container"
Sep 17 19:56:00.006812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006127    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kserve-container"
Sep 17 19:56:00.006812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006176    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kserve-container"
Sep 17 19:56:00.006812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006193    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006203    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kserve-container"
Sep 17 19:56:00.006812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006210    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="247126b2-16da-4af0-a78b-63cdd2565bf3" containerName="kserve-container"
Sep 17 19:56:00.006812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006217    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kserve-container"
Sep 17 19:56:00.006812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006222    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kserve-container"
Sep 17 19:56:00.006812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006228    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006233    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="536aa7c2-08cf-4a18-b895-c4c8997acafb" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006237    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="960b3886-5a54-450d-8584-1e041c1f5c59" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006242    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b188c85-659e-4241-8c59-f2d68a54da68" containerName="kube-rbac-proxy"
Sep 17 19:56:00.006812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006246    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="66961fd2-01c2-48d2-899c-5847a7b53b56" containerName="kserve-container"
Sep 17 19:56:00.006812 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.006252    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8833583-eddf-4ca0-85e4-9ae83fa224d4" containerName="kube-rbac-proxy"
Sep 17 19:56:00.010816 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.010798    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9dgbs/must-gather-8qzdq"
Sep 17 19:56:00.013397 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.013372    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9dgbs\"/\"kube-root-ca.crt\""
Sep 17 19:56:00.014531 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.014502    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9dgbs\"/\"openshift-service-ca.crt\""
Sep 17 19:56:00.014678 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.014597    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9dgbs\"/\"default-dockercfg-lfrpr\""
Sep 17 19:56:00.017255 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.017233    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9dgbs/must-gather-8qzdq"]
Sep 17 19:56:00.086096 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.086057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a151a6a8-e5a5-4b32-9833-4bc1288f499c-must-gather-output\") pod \"must-gather-8qzdq\" (UID: \"a151a6a8-e5a5-4b32-9833-4bc1288f499c\") " pod="openshift-must-gather-9dgbs/must-gather-8qzdq"
Sep 17 19:56:00.086297 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.086128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7pcw\" (UniqueName: \"kubernetes.io/projected/a151a6a8-e5a5-4b32-9833-4bc1288f499c-kube-api-access-d7pcw\") pod \"must-gather-8qzdq\" (UID: \"a151a6a8-e5a5-4b32-9833-4bc1288f499c\") " pod="openshift-must-gather-9dgbs/must-gather-8qzdq"
Sep 17 19:56:00.186651 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.186612    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a151a6a8-e5a5-4b32-9833-4bc1288f499c-must-gather-output\") pod \"must-gather-8qzdq\" (UID: \"a151a6a8-e5a5-4b32-9833-4bc1288f499c\") " pod="openshift-must-gather-9dgbs/must-gather-8qzdq"
Sep 17 19:56:00.186842 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.186674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7pcw\" (UniqueName: \"kubernetes.io/projected/a151a6a8-e5a5-4b32-9833-4bc1288f499c-kube-api-access-d7pcw\") pod \"must-gather-8qzdq\" (UID: \"a151a6a8-e5a5-4b32-9833-4bc1288f499c\") " pod="openshift-must-gather-9dgbs/must-gather-8qzdq"
Sep 17 19:56:00.186964 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.186946    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a151a6a8-e5a5-4b32-9833-4bc1288f499c-must-gather-output\") pod \"must-gather-8qzdq\" (UID: \"a151a6a8-e5a5-4b32-9833-4bc1288f499c\") " pod="openshift-must-gather-9dgbs/must-gather-8qzdq"
Sep 17 19:56:00.195701 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.195672    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7pcw\" (UniqueName: \"kubernetes.io/projected/a151a6a8-e5a5-4b32-9833-4bc1288f499c-kube-api-access-d7pcw\") pod \"must-gather-8qzdq\" (UID: \"a151a6a8-e5a5-4b32-9833-4bc1288f499c\") " pod="openshift-must-gather-9dgbs/must-gather-8qzdq"
Sep 17 19:56:00.337361 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.337332    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9dgbs/must-gather-8qzdq"
Sep 17 19:56:00.608208 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.608184    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9dgbs/must-gather-8qzdq"]
Sep 17 19:56:00.610645 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:56:00.610611    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda151a6a8_e5a5_4b32_9833_4bc1288f499c.slice/crio-86d82920314ca98ea19fd52e8882b028d01f9b7ef940ea7b9ec6536dd202be4f WatchSource:0}: Error finding container 86d82920314ca98ea19fd52e8882b028d01f9b7ef940ea7b9ec6536dd202be4f: Status 404 returned error can't find the container with id 86d82920314ca98ea19fd52e8882b028d01f9b7ef940ea7b9ec6536dd202be4f
Sep 17 19:56:00.612474 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:00.612425    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:56:01.172744 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:01.172707    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9dgbs/must-gather-8qzdq" event={"ID":"a151a6a8-e5a5-4b32-9833-4bc1288f499c","Type":"ContainerStarted","Data":"86d82920314ca98ea19fd52e8882b028d01f9b7ef940ea7b9ec6536dd202be4f"}
Sep 17 19:56:02.181508 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:02.181473    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9dgbs/must-gather-8qzdq" event={"ID":"a151a6a8-e5a5-4b32-9833-4bc1288f499c","Type":"ContainerStarted","Data":"025412b971908684c1b951ab5f7434347a290af864b2b70f5d7c8a060fdb6e5a"}
Sep 17 19:56:02.181508 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:02.181517    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9dgbs/must-gather-8qzdq" event={"ID":"a151a6a8-e5a5-4b32-9833-4bc1288f499c","Type":"ContainerStarted","Data":"dbcf930fb5d4e190800c84358f429f9710bbbe18d6b8cafe6d2aa48e21f5a0e5"}
Sep 17 19:56:02.197046 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:02.196983    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9dgbs/must-gather-8qzdq" podStartSLOduration=2.282742795 podStartE2EDuration="3.19695894s" totalImagesPullingTime="914.216145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:56:02.196154773 +0000 UTC m=+2966.024013028" watchObservedRunningTime="2026-09-17 19:56:02.19695894 +0000 UTC m=+2966.024817178"
Sep 17 19:56:03.035118 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:03.035086    2583 ???:1] "http: TLS handshake error from 10.0.143.217:48826: EOF"
Sep 17 19:56:03.042047 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:03.042019    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x5kmn_6f451007-41b2-4c22-8441-36a2b6284c60/global-pull-secret-syncer/0.log"
Sep 17 19:56:03.200345 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:03.200310    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ggdrv_dee9f420-c08e-4466-80b6-dbb7d50d36ea/konnectivity-agent/0.log"
Sep 17 19:56:03.222128 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:03.222041    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-177.ec2.internal_6ee8c51dd833e290a1526e3b75d4bd28/haproxy/0.log"
Sep 17 19:56:07.093477 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:07.093435    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2524f23-bfbf-4d32-bd23-50a9de492cba/alertmanager/0.log"
Sep 17 19:56:07.135924 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:07.135892    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2524f23-bfbf-4d32-bd23-50a9de492cba/config-reloader/0.log"
Sep 17 19:56:07.165010 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:07.164985    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2524f23-bfbf-4d32-bd23-50a9de492cba/kube-rbac-proxy-web/0.log"
Sep 17 19:56:07.197659 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:07.197621    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2524f23-bfbf-4d32-bd23-50a9de492cba/kube-rbac-proxy/0.log"
Sep 17 19:56:07.231560 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:07.231529    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2524f23-bfbf-4d32-bd23-50a9de492cba/kube-rbac-proxy-metric/0.log"
Sep 17 19:56:07.259127 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:07.259078    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2524f23-bfbf-4d32-bd23-50a9de492cba/prom-label-proxy/0.log"
Sep 17 19:56:07.296530 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:07.296504    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2524f23-bfbf-4d32-bd23-50a9de492cba/init-config-reloader/0.log"
Sep 17 19:56:07.549589 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:07.549555    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5fx8b_69e4e049-b862-482f-a97a-6755cd479291/node-exporter/0.log"
Sep 17 19:56:07.575662 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:07.575625    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5fx8b_69e4e049-b862-482f-a97a-6755cd479291/kube-rbac-proxy/0.log"
Sep 17 19:56:07.602071 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:07.602032    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5fx8b_69e4e049-b862-482f-a97a-6755cd479291/init-textfile/0.log"
Sep 17 19:56:07.832581 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:07.832495    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-krhhj_3f0e9b4b-68cb-4902-832b-5907880fc427/kube-rbac-proxy-main/0.log"
Sep 17 19:56:07.859861 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:07.859834    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-krhhj_3f0e9b4b-68cb-4902-832b-5907880fc427/kube-rbac-proxy-self/0.log"
Sep 17 19:56:07.885768 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:07.885738    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-krhhj_3f0e9b4b-68cb-4902-832b-5907880fc427/openshift-state-metrics/0.log"
Sep 17 19:56:08.190565 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:08.190482    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-w8d54_e6a0e9a3-4309-42a4-bc21-c5131f231ab6/prometheus-operator-admission-webhook/0.log"
Sep 17 19:56:08.230308 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:08.230273    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5668b895b4-x2sh9_29ef5b94-212d-4117-a6b1-38ba1130ad41/telemeter-client/0.log"
Sep 17 19:56:08.273717 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:08.273691    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5668b895b4-x2sh9_29ef5b94-212d-4117-a6b1-38ba1130ad41/reload/0.log"
Sep 17 19:56:08.302192 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:08.302139    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5668b895b4-x2sh9_29ef5b94-212d-4117-a6b1-38ba1130ad41/kube-rbac-proxy/0.log"
Sep 17 19:56:10.830645 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:10.830614    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d7fddb4ff-r9nhq_c3b00832-730b-498a-a79a-fa5b6ca6d7eb/console/0.log"
Sep 17 19:56:10.867248 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:10.867210    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-lcwsh_a4457e63-37ef-4ef9-8cc0-47028302cb94/download-server/0.log"
Sep 17 19:56:11.753523 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.753488    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"]
Sep 17 19:56:11.759724 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.759689    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:11.766578 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.766545    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"]
Sep 17 19:56:11.894805 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.894763    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/988adc98-9bf3-4bd9-bd99-d9cdea54d5c2-podres\") pod \"perf-node-gather-daemonset-pjdsh\" (UID: \"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2\") " pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:11.895274 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.894846    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/988adc98-9bf3-4bd9-bd99-d9cdea54d5c2-sys\") pod \"perf-node-gather-daemonset-pjdsh\" (UID: \"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2\") " pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:11.895274 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.894868    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/988adc98-9bf3-4bd9-bd99-d9cdea54d5c2-proc\") pod \"perf-node-gather-daemonset-pjdsh\" (UID: \"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2\") " pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:11.895274 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.894904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/988adc98-9bf3-4bd9-bd99-d9cdea54d5c2-lib-modules\") pod \"perf-node-gather-daemonset-pjdsh\" (UID: \"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2\") " pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:11.895274 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.894962    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vslzr\" (UniqueName: \"kubernetes.io/projected/988adc98-9bf3-4bd9-bd99-d9cdea54d5c2-kube-api-access-vslzr\") pod \"perf-node-gather-daemonset-pjdsh\" (UID: \"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2\") " pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:11.996367 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.996329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vslzr\" (UniqueName: \"kubernetes.io/projected/988adc98-9bf3-4bd9-bd99-d9cdea54d5c2-kube-api-access-vslzr\") pod \"perf-node-gather-daemonset-pjdsh\" (UID: \"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2\") " pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:11.997215 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.997187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/988adc98-9bf3-4bd9-bd99-d9cdea54d5c2-podres\") pod \"perf-node-gather-daemonset-pjdsh\" (UID: \"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2\") " pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:11.997653 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.997538    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/988adc98-9bf3-4bd9-bd99-d9cdea54d5c2-podres\") pod \"perf-node-gather-daemonset-pjdsh\" (UID: \"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2\") " pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:11.997820 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.997622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/988adc98-9bf3-4bd9-bd99-d9cdea54d5c2-sys\") pod \"perf-node-gather-daemonset-pjdsh\" (UID: \"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2\") " pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:11.998043 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.998028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/988adc98-9bf3-4bd9-bd99-d9cdea54d5c2-proc\") pod \"perf-node-gather-daemonset-pjdsh\" (UID: \"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2\") " pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:11.998313 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.997972    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/988adc98-9bf3-4bd9-bd99-d9cdea54d5c2-sys\") pod \"perf-node-gather-daemonset-pjdsh\" (UID: \"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2\") " pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:11.998454 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.998202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/988adc98-9bf3-4bd9-bd99-d9cdea54d5c2-proc\") pod \"perf-node-gather-daemonset-pjdsh\" (UID: \"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2\") " pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:11.998724 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.998662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/988adc98-9bf3-4bd9-bd99-d9cdea54d5c2-lib-modules\") pod \"perf-node-gather-daemonset-pjdsh\" (UID: \"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2\") " pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:11.998863 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:11.998283    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/988adc98-9bf3-4bd9-bd99-d9cdea54d5c2-lib-modules\") pod \"perf-node-gather-daemonset-pjdsh\" (UID: \"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2\") " pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:12.005324 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:12.005262    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vslzr\" (UniqueName: \"kubernetes.io/projected/988adc98-9bf3-4bd9-bd99-d9cdea54d5c2-kube-api-access-vslzr\") pod \"perf-node-gather-daemonset-pjdsh\" (UID: \"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2\") " pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:12.076431 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:12.076387    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:12.369546 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:12.369501    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"]
Sep 17 19:56:12.374841 ip-10-0-128-177 kubenswrapper[2583]: W0917 19:56:12.374391    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod988adc98_9bf3_4bd9_bd99_d9cdea54d5c2.slice/crio-2f6f19554862198aceac799ac393d2d62269be3e9712404bf9bb2ed76a7d77be WatchSource:0}: Error finding container 2f6f19554862198aceac799ac393d2d62269be3e9712404bf9bb2ed76a7d77be: Status 404 returned error can't find the container with id 2f6f19554862198aceac799ac393d2d62269be3e9712404bf9bb2ed76a7d77be
Sep 17 19:56:12.462287 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:12.462242    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-66n6k_92e4146f-3a14-40d3-bbb5-83bbe2f8be67/dns/0.log"
Sep 17 19:56:12.484890 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:12.484860    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-66n6k_92e4146f-3a14-40d3-bbb5-83bbe2f8be67/kube-rbac-proxy/0.log"
Sep 17 19:56:12.563357 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:12.563325    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kdh4s_f6cc77d6-c367-470c-bfca-7399dc10d65b/dns-node-resolver/0.log"
Sep 17 19:56:13.113790 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:13.113757    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5c588ccb4c-d45nx_fad2458d-4ef5-4d9b-99c4-e323be51944e/registry/0.log"
Sep 17 19:56:13.166168 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:13.166135    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m68t4_68a1d862-0402-4b78-a9fc-d61d8b1816ac/node-ca/0.log"
Sep 17 19:56:13.260290 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:13.260255    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh" event={"ID":"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2","Type":"ContainerStarted","Data":"9a3809f8164b39f50312c715eb44346cf7cf2e0ae4271116a74a9c549d18eba4"}
Sep 17 19:56:13.260599 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:13.260579    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:13.260744 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:13.260726    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh" event={"ID":"988adc98-9bf3-4bd9-bd99-d9cdea54d5c2","Type":"ContainerStarted","Data":"2f6f19554862198aceac799ac393d2d62269be3e9712404bf9bb2ed76a7d77be"}
Sep 17 19:56:13.277349 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:13.277288    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh" podStartSLOduration=2.27726449 podStartE2EDuration="2.27726449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:56:13.276272208 +0000 UTC m=+2977.104130456" watchObservedRunningTime="2026-09-17 19:56:13.27726449 +0000 UTC m=+2977.105122725"
Sep 17 19:56:14.387318 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:14.387291    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vtv4x_14e87624-a049-426c-b0ef-8d21d1100337/serve-healthcheck-canary/0.log"
Sep 17 19:56:14.972502 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:14.972474    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x4c96_b65a92c1-7983-4dc0-afd3-eed290a55ffb/kube-rbac-proxy/0.log"
Sep 17 19:56:14.995387 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:14.995357    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x4c96_b65a92c1-7983-4dc0-afd3-eed290a55ffb/exporter/0.log"
Sep 17 19:56:15.018642 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:15.018613    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x4c96_b65a92c1-7983-4dc0-afd3-eed290a55ffb/extractor/0.log"
Sep 17 19:56:17.414680 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:17.414645    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-pwhcf_91f2b8d6-c5d4-4288-ae95-3a68d992e56a/manager/0.log"
Sep 17 19:56:17.610154 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:17.610123    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-cnljh_121beb01-4aac-48ee-b683-59bc4acc439b/seaweedfs/0.log"
Sep 17 19:56:19.277620 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:19.277545    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9dgbs/perf-node-gather-daemonset-pjdsh"
Sep 17 19:56:22.092836 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:22.092801    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-knwrz_6b111e2f-6f34-4f14-9e41-119a5d6e112b/migrator/0.log"
Sep 17 19:56:22.115343 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:22.115300    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-knwrz_6b111e2f-6f34-4f14-9e41-119a5d6e112b/graceful-termination/0.log"
Sep 17 19:56:23.857065 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:23.857035    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j74f8_a41b66ba-345d-4b30-9987-043f67724e6e/kube-multus-additional-cni-plugins/0.log"
Sep 17 19:56:23.879633 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:23.879605    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j74f8_a41b66ba-345d-4b30-9987-043f67724e6e/egress-router-binary-copy/0.log"
Sep 17 19:56:23.901882 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:23.901850    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j74f8_a41b66ba-345d-4b30-9987-043f67724e6e/cni-plugins/0.log"
Sep 17 19:56:23.923855 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:23.923827    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j74f8_a41b66ba-345d-4b30-9987-043f67724e6e/bond-cni-plugin/0.log"
Sep 17 19:56:23.944059 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:23.944033    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j74f8_a41b66ba-345d-4b30-9987-043f67724e6e/routeoverride-cni/0.log"
Sep 17 19:56:23.964828 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:23.964799    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j74f8_a41b66ba-345d-4b30-9987-043f67724e6e/whereabouts-cni-bincopy/0.log"
Sep 17 19:56:23.986754 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:23.986726    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j74f8_a41b66ba-345d-4b30-9987-043f67724e6e/whereabouts-cni/0.log"
Sep 17 19:56:24.027880 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:24.027810    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jq2zg_3c6753d8-f30c-4345-9c1b-e0aec1d11c24/kube-multus/0.log"
Sep 17 19:56:24.174346 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:24.174318    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r2lm8_d7c98728-9f29-4061-9131-95535a68e5c0/network-metrics-daemon/0.log"
Sep 17 19:56:24.195539 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:24.195508    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r2lm8_d7c98728-9f29-4061-9131-95535a68e5c0/kube-rbac-proxy/0.log"
Sep 17 19:56:24.962401 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:24.962373    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-controller/0.log"
Sep 17 19:56:24.981642 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:24.981614    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/0.log"
Sep 17 19:56:24.981804 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:24.981626    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovn-acl-logging/1.log"
Sep 17 19:56:25.001582 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:25.001537    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/kube-rbac-proxy-node/0.log"
Sep 17 19:56:25.026459 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:25.026415    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 19:56:25.046999 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:25.046971    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/northd/0.log"
Sep 17 19:56:25.068116 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:25.068085    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/nbdb/0.log"
Sep 17 19:56:25.092189 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:25.092157    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/sbdb/0.log"
Sep 17 19:56:25.142858 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:25.142802    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9lk9z_f75cfa47-2a6a-4335-aeb7-2f3fd0fa156c/ovnkube-controller/0.log"
Sep 17 19:56:26.536828 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:26.536798    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-99d9m_29449e74-88a3-41dd-bc8f-042b26ad1055/check-endpoints/0.log"
Sep 17 19:56:26.560584 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:26.560553    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-sv444_f78a6d46-2733-4c1a-83a9-e566098efa57/network-check-target-container/0.log"
Sep 17 19:56:27.719619 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:27.719582    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-csq8c_9a1fa965-ba48-4ea1-993c-11ac62dd09b4/iptables-alerter/0.log"
Sep 17 19:56:28.394485 ip-10-0-128-177 kubenswrapper[2583]: I0917 19:56:28.394458    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-dbvvq_5e7c41cc-0eda-45d2-8acb-b8f93a463a13/tuned/0.log"